News

I will be patching the VMWare server that runs trac, irc, and one of the UCB serial servers. I do not expect this to take more than 30 min.

We are extending the planned downtime for today until 7PM. Sorry for any inconvenience.

We will be upgrading the switch firmware on the Nortel switches to the latest release which should fix some multicast issues introduced with the previous release. This will affect the control network since the switches need to reboot when the firmware is upgraded. You can track the completion of the upgrade by checking this ticket in trac: https://trac.deterlab.net/ticket/124 (login required). We will also be rebooting boss and users during this time.

You are invited to participate in the 3rd CSET (Cyber Security Experimentation and Test) 2010 workshop being held in Washington DC on Monday, August 9th. The Early Bird Registration Deadline is Monday, July 19, 2010 to receive the greatest savings.

Registration and details of the workshop are available at: http://www.usenix.org/events/cset10/index.html

Attention, students! CSET '10 has a limited number of student travel grants available. The deadline for applying is July 15.

CSET focuses on the science, design, architecture, construction, operation, and use of cybersecurity experiments in network testbeds and infrastructures. The workshop's scope includes all work relevant to cyber security experimentation and evaluation including simulation, emulation, deployment, traffic models.

We have an interesting and stimulating program this year, including a keynote address by Dr. Doug Maughan, Program Manager, U.S. Department of Homeland Security's Science and Technology Directorate on "The Role of Testbeds in CyberSecurity Research" along with presentations on Cyber Physical systems, emulation testbeds as well as key sessions on Security Education, Work in Progress and brainstorming sessions. In addition, a discussion on "Security Experimentation with Cyber-Physical Devices" is scheduled with a panel that comprises of key practitioners in the field.

We look forward to seeing you in Washington DC!

DO NOT MODIFY SWAPPED IN EXPERIMENTS

If you want to modify your experiment, swap it out first.

We are working on fixing this ASAP. We apologize for any inconvenience.

We have placed the binary packages released with FreeBSD 7.3 on scratch and have updated root's .cshrc on the FBSD7-STD image to automatically download from scratch. To install precompiled binary packages on FreeBSD 7, simply use the -r switch with pkg_add.

A new Ubuntu 10.04 LTS image is now available. The osid is Ubuntu1004-LTS. Please file a trouble ticket if you run into problems with the image.

FreeBSD 7.2 is going end of life at the end of the month. I will be upgrading users and boss to FreeBSD 7.3. Both machines will be rebooted during the usual Thursday downtime.

the cooling system in the berkeley black box has been repaired. The failure was due to a mixing valve under servo control; the water supplied by the building is too cold, and so it it is mixed with recirculated water. The valve at this location seems to fail and need to be replaced every 6 - 9 months; we now keep a replacement part *on site*, but we are not a liberty to replace it ourselves; only somebody from the central campus maintenance division is authorized to inspect, and either replace themselves or hire a contractor, and the office responsible is not staffed on weekends. At least this time we did not have to wait an extra 3 days to have a spare valve manufactured and shipped to us, but it is almost certain that this will happen again in another 6 to 9 months.

We are having our project review on June 9th and 10th. Any review related demo experiments will be given priority on the testbed. We do not expect interruption of service as we have already allocated the necessary resources to our experiments.

The control network will be interrupted for about 20 min while I downgrade the firmware on Nortel10 to work around a multicast issue that seems to have been introduced with the latest Nortel firmware.

Control Net Separation Testing will be happening Monday and tuesday between 3 and 6pm, Wednesday, thursday and friday between 5 and 8 pm

We have updated the FBSD7-STD image to FreeBSD 7.3. This updated image features larger partitions to make building custom kernels easier and the source for FreeBSD 7.3 is in /share/freebsd/7.3.

The CentOS image has been updated in order to fix a compatibility problem with SEER and to install the latest security fixes.

We will be enabling control network separation on Tuesday May 4th. There should be about a 15 minute interruption on the control network between 1PM and 3PM PST.

There's a significant amount of student travel grants available for IEEE Symposium on Security and Privacy in Oakland, California. This is a premier conference in security and privacy. The eligibility criteria is that one must be a student at a US institution.

For more information and to apply see: http://oakland31.cs.virginia.edu/grants.html. The deadline is April 2 but it may be extended.

The main firewall for DETER will be updated to FreeBSD 7.3 and rebooted during our normal downtime on Thursday, April 1. The downtime should be minimal.

On behalf of the 3rd Workshop on Cyber Security Experimentation and Test (CSET '10) program committee, we'd like to invite you to submit papers on the science, design, architecture, construction, operation, and use of cyber security experiments in network testbeds and infrastructures. Please submit all papers by May 24, 2010, 11:59 p.m. PDT.

Topics of interest include but are not limited to:

• Science of security/testbed experimentation
  • Data and tools to achieve realistic experiment setup/scenarios
  • Diagnosis of and methodologies for dealing with experimental artifacts
  • Support for experimentation on a large scale (virtualization, federation, high fidelity scale-down)
  • Tools and methodologies to achieve, and metrics to measure, correctness, repeatability, and sharing of experiments
• Testbeds and methodologies
  • Tools, methodologies, and infrastructure that support risky experimentation
  • Support for experimentation in emerging security topics (cyber-physical systems, wireless, botnets, etc.)
  • Novel experimentation approaches (e.g., coupling of emulation and simulation)
  • Experience in designing or deploying secure testbeds
  • Instrumentation and automation of experiments; their archiving, preservation, and visualization
  • Fair sharing of testbed resources
• Hands-on security education
  • Experiences teaching security classes that use hands-on security experiments for homework, in-class demonstrations, or class projects
  • Experiences from red team/blue team exercises
  Submissions are due Monday, May 24, 2010, 11:59 p.m. PDT. For more details on the submission process, please see the complete Call for Papers at: http://www.usenix.org/cset10/cfpa/

  We look forward to receiving your submissions!
  
  Terry V. Benzel, USC Information Sciences Institute (ISI) CSET '10 General Chair
  Jelena Mirkovic, USC Information Sciences Institute (ISI)
  Angelos Stavrou, George Mason University
  CSET '10 Program Co-Chairs, cset10chairs@usenix.org.

Developments in cloud computing and ubiquitous network computing have increased the reliability and safety of advanced large-scale network systems, and driven the demand for rapid advances in these systems. Enhancing and upgrading the testbeds used to test network systems has become a necessity.

National Institute of Information and Communications Technology (NICT) and Japan Advanced Institute of Science and Technology (JAIST) will hold "International Symposium on ICT System Testbeds" on March 30, 2010.

This symposium includes lectures by specialists from around the world on international trends and research findings in ICT system testbeds, as well as future prospects and expectations, and examines the future direction for ICT system testbeds in Japan.

For detailed information of the symposium program and participation, please visit http://starbed.nict.go.jp/ictstb.htm We hope you will attend this symposium.

LLNL is looking for summer interns for their Cyber Defender Program. They are particularly interested in people that had DETERlab experience. For more information and to apply please visit https://cyberdefender.llnl.gov/.

We ran into problems with multicast after upgrading the firmware on the Nortel control net switches last Thursday. We were able to work around the problem by enabling IGMP snooping, but snooping may be problematic when used in combination with control network separation. We will be looking further into the multicast issue during this downtime.

In order to address some performance issues I have recompiled the kernel on users to take out some extra debugging features. Users will be rebooted at 6PM PST today.

We have updated all Nortel switches in the testbed to the latest firmware available which is supposed to fix the vlan creation problem.

The control net interface on the ISI gateway was set to down 100mbit after replacing Foundry10 causing periodic performance problems for the UCB nodes (in particular whenever nodes were reloading). We have reconfigured the interface back to 1000mbits and things seem to be working better now.

Tonight during the downtime (and two hours beyond the downtime) we replaced the remaining Foundry switch on our control network with a pair of Nortel gigabit switches. We apologize that the downtime extended beyond the normal 2 hour window.

During today's normal downtime, we will be replacing our last Foundry control network switch with a pair of Nortel switches. The downtime may last a little longer than usual, but we expect the testbed to be operational again by 9PM PST.

We are having issues with these switches and will be rebooting them shortly (at 8PM PST). Sorry for any inconvenience.

If you are using DETER in classes be sure to check our new page covering DETER policies and support for educational use. You will also find there sample class exercises.

We have added a machine with a NetFPGA board into the testbed. It will be primarily used to support some classes, but feel free to contact testbed-ops if you are interested in using the machine when it is free.

We replaced an older 100mbit Foundry switch that served the control network for the pc3000 class machines with a pair of Nortel 5510 gigabit switches. We were initially going to do this on Sunday, but we decided to do 1/2 of the Sunday upgrade during the regular downtime. There will be no downtime on Sunday. Instead the remaining Foundry will be upgraded during the normal downtime next Thursday.

On the evening of Sunday, Februrary 14th I will be replacing two older Foundry 100mbit switches with Nortel gigabit switches. This downtime should being around 7PM and last a number of hours.

The kernels for the frisbee, newnode, and administrative operating systems has been updated to FreeBSD 7.2 in order to allow up to add newer hardware to the testbed.

The Extra Packages for Enterprise Linux (EPEL) has been mirrored on scratch and the appropriate repositories have been added to the CentOS 5 image.

We have moved the chat script onto www.isi.deterlab.net so that users do not have to accept a self signed certificate for wiki.isi.deterlab.net. Sorry for any trouble using the chat feature.

A new BETA CentOS 5 image is available. The image id is 'CentOS5'. CentOS provides us longer support than the Fedora images do. Fedora releases are supported for only 13 months after a new version comes out. On the other hand, CentOS 5 is scheduled to be end-of-lifed on March 31, 2014. We are hoping that CentOS will become a more stable and better supported alternative to Fedora. Currently we have the base packages and updates mirrored locally. In the future we hope to track the Extra Packages for Enterprise Linux EPEL project which provides redhat packages of software that is present in Fedora but not in CentOS.

I heard from Nortel today that the issue plaguing our Nortel switches is expected to be included with a firmware update which will be released in February. Hopefully this will put an end to the problem with vlans not always getting properly created on the Nortel switches.

We are experiencing some problems with our switches here at ISI. You may experience swap in problems.

ISI will be on holiday until 2010-01-04. While we will attempt to respond to issues as promptly as possible, we cannot guarantee problems will resolved until after that date.

Thank you and sorry for the inconvenience.

The certificate for www.isi.deterlab.net has been updated.

We are up and running a more recent snapshot from Emulab. Please let testbed-ops@isi.deterlab.net know if anything seems broken.

We will be updating to an updated Emulab codebase this Friday at 7PM. The testbed will be unavailable during this upgrade.

Last week we fixed the kernel memory leak that was causing users to run out of kernel memory about every two days. If you are interested in the details, you can view the FreeBSD problem report w/patch: kern/140853: [nfs] [patch] NFSv2 remove calls fail to send error replies (memory leak!)

In order to treat some kernel instability problems we will be rebooting users every night at 12:00 AM PST (UTC-8). Access to shell, the web interface, and home directories (within experiments) will be unavailable for the duration of the reboot. If this affects your experiment or logging, please let us know so we can help accomodate you.

We lost a module on Cisco4 which was the connection point for UCB and Nortel10. I have moved the connections to a different module and updated the database, so swap-ins that span multiple machine types should be working again.

It seems that during the transition from FreeBSD 6.4 to FreeBSD 7.2 the resolver stopped using the search path to resolve hosts when they contained a '.' This means trying to do a ssh node.experiment.project would fail saying that the host name lookup failed. Thankfully, there is an option to control this behavior and we have enabled it in /etc/resolv.conf. From man 5 resolv.conf:

                 options option ...

                 where option is one of the following:

                 debug         sets RES_DEBUG in _res.options.

                 ndots:n       sets a threshold for the number of dots which
                               must appear in a name given to res_query() (see
                               resolver(3)) before an initial absolute query
                               will be made.  The default for n is ``1'',
                               meaning that if there are any dots in a name,
                               the name will be tried first as an absolute
                               name before any search list elements are
                               appended to it.

During the scheduled downtime this Thursday we upgraded boss to FreeBSD 7.2. So far things seem to be working ok.

USERS upgraded was to FreeBSD 7.2 since we were having stability issues with 6.4 which were proving hard to track down. We're hoping that either the problem has been fixed or at least we can continue debugging on a more recent version of FreeBSD.

Users has been experiencing stability issues. We are upgrading to FreeBSD 7.2 so that we do not end up debugging an issue that has already been fixed. Please let testbed-ops@isi.deterlab.net know if anything is not working properly post upgrade. Thanks! The upgrade should be complete by 10PM PST Wednesday, November 11, 2009.

An obscure testbed software bug has been preventing WINXP-UPDATE from loading has been found and fixed for the pc3000 machines. Sorry for any inconvenience.

There was an error in the sources.list file pointing to an incorrect repository in the Ubuntu804-STD image. I fixed this and updated both images to the latest packages. Please let us know if you run into any problems with the updated images.

We are rebuilding the scratch server which hosts the fedora packages and ubuntu packages. We expect it to be back online tomorrow.

The switch was fixed. Thank you for your patience.

We are currently experiencing some difficulties with an errant switch that may cause swap ins to fail. This issue is expected to be resolved within the hour. Please stay tuned.

The packages installed on boss and users have been updated. If you notice anything strange, please contact testbed-ops.

The testbed will be unavailable Sunday, September 13th for a few hours during the afternoon for some upgrades. Experiments will continue to run and stay swapped in, but I will be turning off access to the testbed while performing the upgrade.

The packages that were in /share/ubuntu have been moved to our local package mirror, scratch. There is a new sources list in users:/share/ubuntu to update your existing images with.

We had another panic related to NFS exports being modified while in the middle of checking NFS access. I have backported some locking code from the current version of FreeBSD to hopefully address this problem. I will be installing a new kernel during the downtime today.

I think I have tracked down what was causing users to kernel panic and I have filed a FreeBSD problem report about it (133439). It was related to FreeBSD and nfsd not being SMP safe. I have taken SMP support out of the kernel for the time being. Also, there was some other fallout from the panics related to account that I think has now been taken care of. If you notice anything strange, please let testbed-ops@isi.deterlab.net know.

We have been having some problems with the users node of the testbed. There appears to be a NFS bug that is somehow being tickled and causing a kernel panic. I have configured users to run a debug kernel and to savecore when this happens so that we can track the bug down.

We invite you to submit papers to the Workshop on Cyber Security
Experimentation and Test (CSET'09) on August 10, 2009 in Montreal,
Canada. The CSET'09 workshop is co-located with the USENIX Security
Symposium.

CSET '09 is bringing together researchers and testbed developers to
share their experiences and define a forward-looking agenda for the
development of scientific, realistic evaluation approaches for security
threats and defenses; it provides an important community forum for the
exploration of transformational advances in the field of cyber security
experimentation and test.  

While we particularly invite papers that deal with security
experimentation, we are also interested in papers that address general
testbed/ experiment issues that have implications on security
experimentation such as: traffic and topology generation, large-scale
experiment support, experiment automation, etc. We are further
interested in educational efforts that involve security experimentation.
Please see workshop URL for a more detailed listing of topics.

Financial assistance is expected to be available for promising students
to help defray costs of attending this workshop, present their papers,
and become more integrated into this important scientific community. We
believe that attendance to present papers and to interact with
researchers and practitioners in Cyber Security Experimentation and Test
is an important component of students' education and professional
development. Moreover, students' presence at this workshop will enrich
and broaden the range of workshop activities. Procedures for applying
for a student travel grant are forthcoming.


Workshop URL: http://www.usenix.org/event/cset09/

Important Dates

    * Submissions due: May 15, 2009, 11:59 p.m. PDT
    * Notification to authors: June 30, 2009
    * Electronic files due: July 15, 2009


Workshop Organizers

General Chair
Terry V. Benzel, USC Information Sciences Institute (ISI)

Program Co-Chairs
Jelena Mirkovic, USC Information Sciences Institute (ISI)

Angelos Stavrou, George Mason University

Program Committee
1) Paul Barford, University of Wisconsin
2) Andy Bavier, Princeton University
3) Matt Bishop, University of California, Davis
4) Thomas Daniels, Iowa State University
5) Sonia Fahmy, Purdue University
6) Carrie Gates, Computer Associates
7) Alefiya Hussain, SPARTA Inc.
8) Brent Kang, The University of North Carolina at Charlotte
9) Vern Paxson, ICSI
10) Sean Peisert, University of California, Davis
11) Peter Reiher, University of California, Los Angeles
12) Rob Ricci, University of Utah
13) Mark Stamp, San Jose State University
14) Kashi Vishwanath, Microsoft Research
15) Vinod Yegneswaran, SRI International


We hope to see you in Montreal!

CSET'09 Organizers
Terry Benzel (tbenzel at isi.edu)

Angelos Stavrou (astavrou at gmu.edu)

Jelena Mirkovic (sunshine at isi.edu)

Yesterday we upgraded boss and users to FreeBSD 6.4. So far everything seems to be working without any problems.

Kevin will be testing control net separation again tonight from 5PM until 7PM.

From 5PM to 7PM this evening, we'll be testing out a new control net separation scheme designed to ensure that all experiments are completely isolated from other experiments, even on the other control network. Our hope is that this change will be completely invisible to users (barring some downtime while we reboot the boss node), but please let us know at testbed-ops@deterlab.net if you see problems.

Over the next few weeks we will run a similar series of tests until we finally install the separation scheme as a permanent part of the testbed.

We now have a DETER IRC channel. For more information, go to https://www.isi.deterlab.net/deterchat.php3

There will be a quick downtime tomorrow morning at 7am PST. I will be swapping out a bad memory DIMM in users. The downtime should not last more than 10min.

Our upstream switch here at ISI was replaced this morning and they are still in the process of configuring it. The link to Berkeley is down at the moment and expect intermittent connectivity problems to ISI.

We are hosting monthly phone conferences for DETER users to ask questions of the staff, swap issues and solutions, and look for collaborations. All registered DETER users are cordially invited.

The next user call will be on January 10, 2008, 11 am - noon PST. We will send a reminder and agenda one week before the call. Summaries of previous calls will be found at http://www.isi.edu/deter/telecons/telecons.html.

Join us in Boston, MA, August 6–7, 2007, for the DETER Community Workshop on Cyber Security Experimentation and Test 2007. This workshop will address issues in the design and use of moderate-to-large scale network testbeds to conduct experiments on security topics such as worm propagation, infrastructure defense (e.g., defending the DNS and BGP routing), and denial of service defense. Such experiments are challenging because of complexity, scale, and possible risk.

http://www.usenix.org/events/deter07/

A 200 node experiment has been scheduled for the week of December 18th. The ISI side of the testbed will be unavailable during that time. The week of December 18th is *this week*.

There are a number of Berkeley nodes free for use. To specifically use Berkeley nodes, in your .ns file you can request

tb-set-hardware $node bpc2800

(or bpc3000, or bpc3060)

It is worth noting that lilo based images are not transportable between bpc2800's and anything else; for image compatibility, the 4 types pc3000, pc3060, bpc3000, bpc3060 are essentialy identical.

We just upgraded DETER to FreeBSD-6.1 and incorporated a series of improvements from Emulab. Please report all problems (and there no doubt will be some!) to testbed-ops@isi.deterlab.net.

We have recently fixed a misconfiguration of the DETER testbed that was causing idle experiment detection to fail. Our Cisco and Nortel switches were generating periodic proprietary Ethernet packets, which were registering with the idle system's network traffic counters.

Now that this is fixed, experiments will start to get swapped out after a period of time with no network, tty, or CPU activity. If you wish for your experiments to remain swapped in past the idle time, you can adjust the experiment metadata to prevent idle swap.

For more information, please consult our node use policies.

Tuesday, August 15 and Wednesday, August 16, DETER will be down in order to switchover to a new UPS. We expect to power the systems down Tuesday evening around 10PM, and hope to have them back up sometime after 9AM.

Our expectation is that you can leave your experiments swapped in, and your nodes should come back up when we apply power to the testbed. It would be a good idea to do a 'shutdown -h' on your experimental nodes, to cleanly shut down the systems. As with any operation like this, though, there may be further problems.

In case of problems, Keith Sklower is setting up www.ucb.deterlab.net to mirror the current contents of the ISI machines, so that users could swap in experiments at UCB if necessary.

After some tweaks to the ISI-UCB interconnect, the bpc2800s are now available again from www.isi.deterlab.net.

DETER experienced an unexpected power outage the morning of Saturday, June 24. It scrambled some switch configurations which took us some time to track down and fix. The testbed should be working again now. Please send mail to testbed-ops@isi.deterlab.net if you see problems.

Due to some ongoing difficulties with the bpc2800s, we've decided to make them available via the UCB users node, www.ucb.deterlab.net. They will, at least for the next few weeks, not be available via www.isi.deterlab.net.

Keith Sklower is rsyncing the files from the ISI systems onto the UCB systems, so that users should be able to log into the UCB systems with no problems. Please be aware, though, that future rsyncs could overwrite the files stored at UCB, so be careful.

UCB staff have managed to significantly improve the robustness of the serial connections to the bpc2800s, so serial console access should be much improved.

Keith Sklower of UC Berkeley has added another 30 nodes to the testbed. These bpc2800 nodes will show up as bpc001 - bpc030.

Due to excessive noise in the serial lines, the consoles for these systems have been turned off. You can still run 'console bpc001', but you won't see any output.

As with all Berkeley-based nodes (type bpcxxxx), users should remember that the link between these nodes and the ISI nodes has limited, unpredictable bandwidth, and that this can sometimes effect the speed and reliability of node image loading as well.

DETER will be down both Monday, May 15, and Tuesday, May 16, from 2PM to 5PM, for malware experiments. For safety, the testbed (including users.isi.deterlab.net) will be disconnected from the Internet, and all active experiments will be swapped out.

We've got 64 new Dells, similar to the 64 pc3000s already available at DETER, with dual 3.0 GHz Xeon CPUs, 2GB of RAM, and 36GB 15,000 RPM disks. The new systems have larger CPU caches.

62 of the systems have six network interfaces (five experimental interfaces), and are listed on DETER as pc3060s. The two pc3100s have 10 interfaces (nine experimental) to allow for experiments with more complex topologies.

These systems are connected via Nortel switches -- the experimental switch is a stack of seven Nortel 5510s and one Nortel 5530 with dual 10Gb uplinks, while the control switch is made up of two Nortel 5510s. This is similar the switch configuration for our pc3000s and for the Berkeley nodes.

In addition, Keith Sklower reinstalled the Emulab software with a number of his fixes as well as bug fixes from Utah.

As mentioned in the weekly report #59, experiments have the limitations on the number of links that cross switches and sites. It is due to the way the 'assign' script in the emulab software works. The 'assign' script allocates 100 Mbps per link even though the ns file specifies the link speed as less than 100 Mbps (e.g., 1 Mbps). This means, the number of VLAN's that cross two switches like, Cisco and Nortel at ISI would be limited to 10 even when each link has 1 Mbps as the speed.

To increase the number of VLAN's across the switch boundaries and the two campuses, we set the inter-switch trunk speeds in the testbed database as 4 Gbps, instead of the current real 1 Gbps, so that total of 40, instead of 10 VLAN's can be assigned. The ISI-UCB tunnel speed is set to 1 Gbps, instead of the real 150 Mbps. Doing this would potentially over-subscribe the trunk or the tunnel.

http://www.isi.deterlab.net/doc/Inter-Switch-Bandwidth-Limit.pdf

In the diagram on the above URL, the numbers next to the arrows stand for the actual bandwidth of the links, and the numbers next to them in parentheses are the number of VLAN's that can be assigned over the links. Please keep in mind that these links are shared by all the experiments running on the testbed.