FBI Says Military Had Counterfeit Cisco Routers

There are new developments in the case of the counterfeit Cisco routers, which we have been discussing for some time. The NYTimes updates the story after an FBI PowerPoint presentation made its way onto the Web. It seems that experts at Cisco have examined some of the counterfeit routers in detail and proclaimed that they contain no back doors. Others don't believe we can be so sure. "Last month, [DARPA] began distributing chips with hidden Trojan horse circuitry to military contractors who are participating in the agency's Trusted Integrated Circuits program. The goal is to test forensic techniques for finding hidden electronic trap doors, which can be maddeningly elusive... The threat was demonstrated in April when a team of computer scientists from the University of Illinois presented a paper at a technical conference in San Francisco detailing how they had modified a Sun Microsystems SPARC microprocessor... The researchers were able to create a stealth system that would allow them to automatically log in to a computer and steal passwords."

And outsourcing....

Verification of the producer is essential here - and this is perhaps the moment where outsourcing will bite us in the ass. While you can only buy american made cisco routers, there is no doubt some chipsets made in it are manafactured overseas.

Re:And outsourcing....

While essential, it's not the only step. Automatic tests of the router hardware, random checking, and employee control are all necessary steps if we really want our government networks to be secure.

Re:And outsourcing....

Even automatic checking and testing can be subverted by a determined adversary. For example consider the program, approved by President Reagan and beginning in 1982, whereby the CIA arranged for the Soviets, who were actively attempting to acquire western technology and hardware, to receive natural gas pipeline software and equipment that was designed to "go haywire" after a set amount of time in normal operation. When the pipeline software and hardware, which would have appeared totally normal at first even if the Soviets had bothered to test it, eventually went haywire (i.e. it ran the ultra high pressure scenario) the resulting explosion was so large that it was detected by satellites designed to monitor nuclear explosions from space. The following article [msn.com] from the Washington Post describes this and other programs and operations that took place during the Cold War as part of a coordinated CIA effort to slip bad technology to our enemies.

Re:free software distributes the effort.

Items with high capital costs don't work well as "open source;" basically, the manufacturing plants costs so many billions of dollars that no one who isn't doing proprietary work could afford it. Even if you could open source chip design (a dicey proposition, since there are many fewer EE Phds that want to donate time than there are CS Phds,) there are still difficulties with the actual manufacturing, and we would still need to guarantee the physical chips, which are individual, and cannot be "re-compiled;" if you think there may be an issue with a batch, you can't start over without paying for new chips.

Maybe, however, I am missing something about the procedure you are proposing; what parts would be open source?

Re:

This is all coming down to the fact that we need to assume NO network is secure; that we may be subject to man-in-the-middle attacks even within our own networks.

The solution is not to verify every chip, because that's probably impossible. Somebody's going

Re:

I've heard that the problem is a lot more severe than they're actually letting on. Apparently, one of the biggest problems that they're facing is counterfeit Microsoft products. Fortunately, they're pretty easy to recognize, you just have to educate your e

Re:And outsourcing....

I would agree on this one 100%. Outsourcing is in part responsible for this, although, we can't ever be 100% sure of goods manufactured domestically. These companies could care less about counterfeiting Cisco routers so let's say Cisco were to pull the contract. What's to stop the outsourced producer from selling these counterfeits in their own country without the Cisco name? Maybe they already are because presumably (as I understand) because IP law is hard or impossible to enforce in some countries.

As the NSA already seems to be certifying comm. gear in the military (or might even make the chips for it). Perhaps even for other departments like the FBI. I see one possibility of this that the NSA certifies routers (or makes them itself) or at least makes them in the USA. I don't work with routers nor am I familiar with their manufacturer. I guess my last point, pertaining at least to the FBI investigation, would be invalid if Cisco makes some routers in the USA except, as you indicate, for some chipsets. Though even on chipset in itself could pose a significant risk.

I'm just surpised that the FBI is even making a "presentation" to anyone on this; regardless of wether the presentation leaked or not.

Re:

As you say, even domestically-produced hardware can theoretically have trojans in it, so it should be standard practice to certify everything they use, regardless of where it came from.

The outsourcing boogeyman has nothing to do with this - relying on the

Re:

It's worth noting you can do everything a Cisco router can do with a Linux box. I just built a box with Zebra and a solid state hard drve along with a 4 port network card. I have some pretty good throughput with that and I would have no trouble adding addi

Re:And outsourcing....

It's worth noting you can do everything a Cisco router can do with a Linux box.

Except connect to a SONNET network. Or a DS3 interface. Or aggregate multiple T1s. Or suport terabit switching and routing speeds.

Re:

SONET, DS3, and DSUs in any quantity can be purchased for a computer without a problem. When you get into the terabit range you still have a lot of options.

For switching Cisco makes zero sense, HP gives you higher through-put for less money and they aren

Re:

For some applications, it might be better to sacrifice performance and cost, and implement most of the logic in FPGAs. Then only one chip needs to be verified, and it can be bought from lots that were fabricated before you even finished your design.

Re:And outsourcing....

I don't know if that will be enough. I remember there was a story from the Cold War on how the CIA spied on the KGB. The KGB used Xerox copiers in their offices. I don't know if the CIA convinced Xerox to modify the copier or they modified it before the KGB received it, but the copier would record all copies to flash memory. Every so often, the CIA would have to retrieve the memory. The KGB eventually got suspicious that one machine seemed to be serviced all the time while the other one wasn't. They weighed both machines and found a tiny difference in weights. Eventually they found an extra board. That's my recollection of it. I can't seem to google for the backstory. Even if you bought 100% American parts, there is not guarantee that it wasn't tampered with during a routine repair and maintenance or tampered with in the manufacturing process.

Re:And outsourcing....

http://www.parascope.com/articles/0197/xerox.htm [parascope.com]

Re:

but the copier would record all copies to flash memory

Flash memory... cold war? Surely you must be joking ...

They used a camera with a roll of film, which they then had to develop ... whippersnappers! get off my lawn!

We've always been at war with Eurasia

Somehow, I find it hard to believe that DARPA INTENTIONALLY planted vulnerable chips into potentially critical military systems.

This sounds like a case of spin worthy of Winston Smith from the Minstry of Truth.

Re:

I merely skimmed one article which said that Cisco examined the routers and found no backdoors. The Ministry of Peace is probably just trying to test the sneakiness of their own snooping electronics in the name of "national security". The trojans which are

Re:

* Clarification: "trojans" in the hardware or firmware as well as the software sense.

Re:

okay... that last post had my mind goind way down the wrong path... "Cisco examined the routers in depth, and didn't find any trojans, only a few durex wrappers"

"Counterfeit" not an issue...

From what I understand, the counterfeit routers are made in the same factories by the same people who make the real routers; they just keep the assembly line running past the hours that Cisco is paying them for.

In this case, if Cisco is comparing the counterfeit routers to their legit ones, they should always be the same.

The question this doesn't answer is this: does the LEGIT Cisco equipment contain back doors? How can Cisco be sure it doesn't? Most of the components are manufactured offshore and the assembly is done offshore. Have they examined each part with an electron microscope to verify it doesn't do anything more than what the spec says it should do?

They can't just watch for network activity; these routers might be filtering and caching data waiting for the eventual physical removal of the router in the next upgrade cycle -- or, they might all have a kill switch built in, so someone can remotely take out ALL routers. There are an infinite number of possibilities to look for, and since Cisco doesn't manufacture everything in-house, they really don't have much hope of detecting that none of the infinite possible modifications have been made.

"Partnership"

Anne McCaffrey wrote a book called PartnerShip [amazon.com] with a plot very similar to this situation. The villian provides chips to the Galaxy, including the military. When nearly everyone has upgraded, it turns out that he can remotely control every device, including military hardware, controlled by the chips. That's enough of a spoiler. How can such a grand and well planned scheme be defeated? You'll have to read to find out...

Re:

Great Book, great series of books, though it's been a long time since I've read them.

If you're looking for more: "This ship who *" and "The city who fought"

Re:

Simple: One of the big carriers is too old to be fitted with the new chips as it faces decommission as a museum. When the bad guy shuts down almost all ships in the galaxy, this old ship is reactivated and sets out along with the few ships that weren't des

Re:"Counterfeit" not an issue...

I am generally for free trade and against protectionism, but I am leaning more and more towards the need for a law that makes it mandatory that all gear (guns, routers, computers, coffee makers, etc.) purchased by the Government for any use that is even remotely sensitive be made in the US by US owned companies. That won't necessarily solve this kind of problem, but it would certainly make it far easier to prosecute entities who do things that threaten our national security.

Government purchasing

I am generally for free trade and against protectionism, but I am leaning more and more towards the need for a law that makes it mandatory that all gear (guns, routers, computers, coffee makers, etc.) purchased by the Government for any use that is even remotely sensitive be made in the US by US owned companies. That won't necessarily solve this kind of problem, but it would certainly make it far easier to prosecute entities who do things that threaten our national security.

As for "prosecuting" the military has weapons for that sort of thing. Lot cheaper to send a team of Navy Seals to handle a situation than to insist everything be US made.

On a more serious note, I think you should take some time to look at how the US gove

Re:

As for "prosecuting" the military has weapons for that sort of thing. Lot cheaper to send a team of Navy Seals to handle a situation than to insist everything be US made.

Ha. Yeah. Let's send the military after, say, China where a significant amount of th

Re:

And that software and hardware should open-sourced. This will, hopefully, allow peer review for security, snafu, and waste. Granted, this may not be practical in all situations. However, I think the Linux/BSD community has shown that peer reviewed and c

Re:"Counterfeit" not an issue...

As being someone who recently has purchased several Cisco products on Ebay lately, I can tell you that the counterfeit items are not made on the same assembly line. There are several design differences between them.I use the "Andover test" to tell if I'm purchasing authentic Cisco cards.

If I did purchase a card or Cisco product that did pass the Andover test, then chances are that it was manufactured on the same assembly line, but then you would most likely see a report of a duplicate mac address on a "genuine" Cisco product somewhere. So yes it's a possibility, but highly unlikely IMHO.

Selling out the back door

the counterfeit routers are made in the same factories by the same people who make the real routers; they just keep the assembly line running past the hours that Cisco is paying them for.

That happens ALL the time. I've visited manufacturing plants in China and I've seen it happen with my own eyes. Selling out the back door is not surprising at all. In fact this is why I'm less worried than I might otherwise be about the gear having back doors or being otherwise compromised. Simplest explanation is just theft in one form or another.

does the LEGIT Cisco equipment contain back doors?

Very good question. Got to be worrisome to the US military and security agencies. Much/most off the shelf hardware is made outside the US where it wouldn't me much of a stretch to imagine backdoors have been added by foreign governments. Same worries that other governments have about US made/designed software and hardware. And of course if you really want to get tin-foil-hat about it one has to wonder if our own government has had back doors installed. Very unlikely to be sure, but clearly possible.

That said, it's pretty low on the list of likely threats. Pretty hard to know exactly what gear will be placed where and what it will give you access to. Plus even with a back door, places with sensitive data are more likely to be monitoring the traffic which is harder to hide.

Re:

Pretty hard to know exactly what gear will be placed where and what it will give you access to. Plus even with a back door, places with sensitive data are more likely to be monitoring the traffic which is harder to hide.

They can't just watch for network activity; these routers might be filtering and caching data waiting for the eventual physical removal of the router in the next upgrade cycle -- or, they might all have a kill switch built in, so someone can remotely take out ALL routers. There are an infinite number of possibilities to look for, and since Cisco doesn't manufacture everything in-house, they really don't have much hope of detecting that none of the infinite possible modifications have been made.

Re:

They can't just watch for network activity; these routers might be filtering and caching data waiting for the eventual physical removal of the router in the next upgrade cycle

Which presumes the entity making the modifications has access to the device and the upgrade schedule - rather a stretch I think. Plus, our government isn't exactly known for rapid upgrades. Timeliness of any information would be a huge issue.

-- or, they might all have a kill switch built in, so someone can remotely take out ALL routers.

Which pres

Re:"Counterfeit" not an issue...

From what I understand, the counterfeit routers are made in the same factories by the same people who make the real routers; they just keep the assembly line running past the hours that Cisco is paying them for.

I keep hearing this. But look at the images of the hardware side by side [cachefly.net] ... Is it the same? No it's not. Clearly these two boards are not from the same manufacturing line.

Not a big surprise.

I work for a company that sells used electronics on eBay. We'll occasionally buy cheap gear over eBay too, then resell it at a profit. For many months now we've had a huge problem with counterfeit Cisco cards. It's amazing how detailed the counterfeiters are. My boss wrote up a detailed guide on how to spot fakes. Google "counterfeit cisco wic".

Question is...

... of the DARPA-hacked routers were any of the 'cisco experts' able to determine tampering?

That seems like a logical test, so I have to wonder if they have done it already... or not?

If they contain no backdoors, *THAT WE CAN FIND*, do we continue using th

Fear Fear Fear

Be afraid. Be very afraid. Vote for those that seek to protect you.

This seems like a scare tactic to "warn" people about the dangers of fake hardware/software. Expect a big push around these types of "stories" as more bills like PRO-IP go through congress and as the creation of the IP & Copyright Czar in the Whitehouse gets a big push.

It's a concern but seems to point more to incompetence rather than some difficult-to-spot threat. Why are government agencies not buying directly from Cisco? Seems they should have some sort of corporate connection.

"We must protect our precious bodily fluids."

Re:Fear Fear Fear

1) This has nothing to do with IP rights.

2) It's a concern when you consider the potential effects of this kind of infiltration. Buying directly from Cisco, in no way, protects you from this problem. The hardware is still made overseas in some factory by a bunch of people who may not like the US very much (which is true of 99% of the planet right now).

Apparently you lack the imagination to see how ugly this can get. Fortunately DARPA isn't run by you.

Re:

I wouldn't be so quick to dismiss the IP rights issue. Counterfeiting is all about IP rights. The law doesn't differentiate between you producing knock-off fashion items, work-alike network gear, or burning copies of a music album or movie. Expect these

/Light Bulb Flashes Overhead

So that's why my crappy Linksys wifi access points have to be rebooted every week or so. Damn commies!!!

Re:

No, that's because of the new vxworks firmware.

You reap what you sow

CIA slipped bugs to Soviets [msn.com]

Technical details of malicious hardware

For those of you who are interested, you can find more technical details of how we designed and implemented malicious hardware from here [uiuc.edu]

-- computer scientists from University of Illinois

it is important to carefully inspected new gear.

if your new rack mount routers and switches say "crisco" on the front you may have a problem.

How many back doors? Who has the keys?

The question is not whether Cisco routers have back doors. That has to be assumed. If I was running NSA over the last several decades, I would have my people deep inside every communication equipment manufacturer. The manufacturers management might not even know about it.

The NSA surely has arranged to have one or more back doors designed into virtually every kind of communications switch. The only Cisco employees who would know about them would be the NSA people who work inside Cisco, and some regular Cisco employees who have been cleared. If this has not been done, the NSA senior managers should be fired or jailed.

The real questions are: How many back doors are there? and who has the keys? The (assumed) NSA back door might not be the only one. There is a possibility that the Chinese or Indian chip-fab or software contractors have also installed back doors for their own governments.

With billion-gate machines, a few thousand extra gates would be hard to see. If the extra logic looks like instruction-cache, but just has a little extra code, it would be almost impossible.

the real thing probably also has back doors--ours

I'm certain that if the Chinese haven't in fact installed back doors in bogus (or even real) Cisco routers that they manufacture, they at least have contingency plans for doing so. Their intelligence service wouldn't be doing their job properly if they had

Re:

More like any company that outsources and doesn't perform internal quality control of what they are reselling should be made criminal in this instance of reselling to governmental agencies. Buy a Cisco, throw it in a private LAN sandbox, fire up Wireshark.

Re:

Lets see. A non free society that can barley feed its people now. That has a huge number of people that is now comming into the industrial age and is going to NEED all the energy it can get its hands on very soon is an enemy to be to all who are near.

Re:

two groups of brutal tyrants

I find a considerable amount of what RMS has to say at least thoughtful and challenging, except on political topics.

Re:

Emphasis mine:

an irreparable infrastructure and horrific civil war. If that's not bad enough for you

Is the goal here to trade examples of hyperbole, or to engage in a thorough critical analysis of some arguably crappy policy decisions and tragedies that r

Re:

If you think the US a tyranny, then I wish you could go live in an actual tyranny, briefly, for comparison.

arbitrary or unrestrained exercise of power; despotic abuse of authority. - check! It's just in other countries. the government or rule of a tyrant or absolute ruler. - check! The executive branch has been heading towards full dictatorial powers and can

Re:

Everything you own can be confiscated for suspicion of "making available" crappy RIAA music that can be found on any radio station. Your email, web browsing, phone conversations and church can all be monitored without a warrent [etc]

You know twitter, my