From: "Adam Rothschild" To: nyiix@telehouse.com Sent: Friday, March 9, 2012 9:46:24 PM Subject: Re: NYIIX: ISPnet peering at NYIIX Adding to the below, I'd be interested in seeing a detailed postmortem, with a focus on steps Telehouse is taking to make sure this kind of customer [mis]configuration will never, ever, take down NYIIX in the future. Unfortunately the information out of official channels has been suspiciously sparse thus far. Blaming this all on a software fault in Brocade's IronWare is disingenuous to customers, and stops short of painting a full picture of the problem set. A common best-practice among exchange operators these days is enabling port security, along with the filtering of keepalive/discovery protocols and other possible L2 noise. In addition, sniffers/collectors exist to look for any leaks coming out of customer ports, which can then be quickly traced back and remediated. Hopefully this is an architectural optimization we can count on seeing in the coming days/weeks, though absent any commentary, it's difficult to even speculate. I also remember some stated plans of moving away from a flat VLAN and towards a VPLS-based topology, benefits being better route protection and separation of control and data-plane traffic. Is this still something we can count on, and when? In line with the below, we made the executive decision to disable our NYIIX port for a ~24 hour window following yesterday's issues, in hopes of ensuring platform stability. In the future, peers can expect a similar or lengthened response. -a (Voxel / Internap, AS 29791, http://as29791.peeringdb.com) On 3/9/12 8:36 PM, "Ken Cen" wrote: >We have clients reported data corruption - ie, files sent over the NYIIX >was corrupt. > >What would it take the NYIIX to provide high availability switching >fabric? If it is already commissioned, why wasn't it working? > >Thank you > >-Ken > >-----Original Message----- >From: Bob Tinkelman [mailto:bob@tink.com] On Behalf Of ISPnet Peering >Desk >Sent: Friday, March 09, 2012 7:19 PM >To: nyiix@telehouse.com >Subject: NYIIX: ISPnet peering at NYIIX > >We are ISPnet (as22691). > >All our bgp sessions at the NYIIX are (intentionally) down. > >When last night's Telehouse NYIIX problems caused significant problems >on our backbone switches, we shut down our NYIIX link. > >We hope to be able to turn the port back up within 2 weeks, but will >most likely delay until either we are convinced that Telehouse has fixed >the underlying problem or until we implement a method to prevent any >such reoccurance from propagating to our net. > >We'll email again when we have a firm schedule. > >In the mean time, if any other NYIIX members were able to determine what >sort of traffic was being emitted from the nyiix switches during last >envening's problems, I'd appreciate learning what you know. I've asked >Telehouse but, so far, have not received a response. > >-- >Bob Tinkelman >ISPnet, Inc. http://www.ispnet.net >+1 (718) 464-4747 office >+1 (800) 806-NETS toll free >+1 (718) 217-9407 fax