|Migration starts on Feb 20|
|Wednesday, February 15, 2023, 4:37PM; posted by jelena.|
As promised, we are sharing instructions about user migration to our new infrastructure. The migration will start on Monday, Feb 20. At this point, the migration is voluntary. Please note that all research users must migrate by May 1st.
Migration instructions are now in the DeterLab's documentation (under Support) and a direct link is here: https://docs.deterlab.net/support/migration/. We will send another email on Monday to announce the migration start.
Project leaders should migrate their projects, or delegate this duty to one project member. Project members and project leaders should also migrate their user data.
Please contact us if you have any questions, or you can put in a DeterLab ticket.
|Deterlab is Moving!|
|Tuesday, January 31, 2023, 10:57AM; posted by jdbarnes.|
Dear DeterLab research users, We have some great news to share with you! DeterLab is evolving to new hardware and software, to continue providing stable and robust services to our research community. The new modernized testbed will provide you with significant new experimental capabilities, while offering a smooth process to transition your research projects. This effort has been funded by the NSF's CCRI award #2016643. The modernization of the hardware will increase the lab’s capacity and capabilities. Modernizing its management software with the ISI-developed Merge software will provide support for large-scale, virtualized experiments with reconfigurable topologies. Merge software has powered DARPA’s DCOMP and LightHouse testbeds. ISI has now brought this technology to your fingertips. On February 15th, we will start migrating RESEARCH projects and users to the new platform. We will ask for volunteers first, but please note all research users must migrate by May 1st, 2023. After this date, we will only support CLASS users on the old infrastructure. Our goal is to decommission the old hardware and software during the summer of 2023, so starting Fall 2023 everyone would be on the new infrastructure. Here is a brief summary of what you should expect: - we will provide automated scripts to migrate your experiments and project data to the new infrastructure - UI and support documentation has changed. You can take an early peek here: https://mergetb.org/docs/ - experiment creation process has changed - it is now faster, and less error prone, but it uses a new topology description language - you will be able to request VMs instead of physical machines (you can still have physical machines if you prefer) - you will experience lower resource contention and faster experiment set-up time - the internals of your experiments (e.g., any scripts you currently have to run experiments) will likely require only minor changes to work on the new platform - ticket creation process will transition to email, and we will support chat for quick questions This is an FYI message to help you plan for this transition. We will send a follow-up message on February 15th. Feel free to put in a ticket or reply to this message should you have any questions or concerns. Best regards, The DeterLab team
|Berkeley nodes permanently down|
|Monday, February 7, 2022, 11:35AM; posted by jdbarnes.|
The Berkeley Deter Lab site will be taken offline due to the retirement of our site administrator at Berkeley. Thank you for your understanding.
|Ubuntu18-STD to become default, test your experiments|
|Thursday, December 16, 2021, 12:29PM; posted by jdbarnes.|
As Ubuntu 16.04 is past the End of Support (EOS) date, it will become increasingly difficult to maintain any modern tools on that system.
|Python/pip local mirror|
|Monday, February 1, 2021, 3:00PM; posted by jdbarnes.|
There has been a sustained high volume overuse of pypi web search interface, which has resulted in pypi turning off search for python packages.
Due to this, and to some requests for an available internal mirror for python packages, there is now a static mirror of most python packages at "https://fbsd-build.isi.deterlab.net/pypi/web/simple".
To use this internal mirror, the pip installation stream looks like "pip install -i https://fbsd-build.isi.deterlab.net/pypi/web/simple [package_name]".
We will attempt to get a searchable index running soon.
|Ubuntu1804-STD end-node traffic shaping|
|Thursday, November 5, 2020, 11:02AM; posted by jdbarnes.|
The end-node traffic shaping feature has been debugged and enabled for the supported Ubuntu1804-STD image.
|Heavy usage considerations|
|Thursday, November 5, 2020, 9:38AM; posted by jdbarnes.|
Please try to limit your use of the users.deterlab.net host's bandwidth as it is a shared resource.
If you write files in your experiment, please write them to local disk, then move to users (that is, your home or project directory) once your experiment is completed. Your home directory is NFS mounted in all experimental nodes, so writing files there during an experiment will utilize bandwidth.
If you need to do large (30+ nodes) multi-threaded orchestration, please consider using an existing node within your experiment for this, or allocating an additional node for control of your experiment.
|System Release Announcement: Containers v1.1|
|Monday, June 20, 2016, 1:45PM; posted by alba.|
All components in the DETERLab Containers System have been upgraded as of June 15, 2016. The host OS for all containers is now Ubuntu 14.04. The QEMU-KVM package was upgraded to 2.51-1 and all OpenVZ components were upgraded to the latest versions.
Virtual Distributed Ethernet (VDE) was dropped as the QEMU-KVM network implementation and replaced with a Linux bridge/tap mechanism. This gives an order of magnitude speed increase for QEMU-KVM networking from roughly 450 MBs to 4.5 GBs.
To find out more, go to: http://www.deter-project.org/news/system_release_announcement_containers_v11
|System Release Announcement: MAGI v1.90|
|Friday, June 17, 2016, 4:53PM; posted by jelena.|
The new MAGI version was released on June 6, 2016, and is available to use on the DETERLab testbed. The new release includes support to develop agents in C. The C-agent library includes support for data management, agent response triggers, and logging. To find out more, go to: http://www.deter-project.org/news/system_release_announcement_magi_v190
|Berkeley tunnel end upgraded to 10 gig.|
|Saturday, October 10, 2015, 2:28PM; posted by sklower.|
The Berkeley end of the intercampus tunnel was upgraded to 10 gig on Saturday
The ISI router and endpoint will similarly need to be upgraded before full use of it can be made.
Please expect additional downtimes where experiments will be frozen in place, but we hope with only limited disconnection time.
|Expected Downtime August 17|
|Thursday, August 13, 2015, 9:38AM; posted by jross.|
On Monday, August 17th, we are expecting a power interruption while a faulty UPS is replaced. This power interruption will bring down boss, users, and nearly all of the new dl380g3 nodes.
|Monday, July 20, 2015, 6:43PM; posted by jross.|
We are back up and runnning! Thanks for your patience.
|power loss; swapping locked out|
|Sunday, July 19, 2015, 12:29PM; posted by jross.|
We experienced a complete power loss at ISI and have disabled swapping until we have everything back up and running correctly.
|Saturday, March 14, 2015, 12:01AM; posted by jross.|
An unanticipated dependency on a switch which we removed cost us some troubleshooting time, but the testbed is now back and ready for your experiments!
|Control Net Interruption|
|Friday, March 13, 2015, 9:46AM; posted by jross.|
Some control net traffic had a legacy routing through a switch which we have removed. We are working to re-route this traffic. In the meantime testbed functionality is impaired. We recommend waiting until we have posted that this is resolved before attempting to swap or modify any experiments.
|Downtime Alert! DETER buildout begins March 13|
|Friday, March 6, 2015, 2:15PM; posted by jross.|
Starting on the morning of March 13th, we will be upgrading the DETER testbed with new nodes and switches. Importantly, the pc3000 and pc3060 nodes are being replaced with more modern hardware. The pc3000 and pc3060 nodes will be shut down, so any experiments using them will need to be swapped out before the upgrade begins. Testbed capacity will be heavily impacted by this for the week which we are expecting this buildout to take, so please plan accordingly.
|Testbed Crashed -> UPS failed 3/3/15 10:30 PST (3/4 6:30 UTC)|
|Wednesday, March 4, 2015, 1:08AM; posted by sklower.|
All the servers went off the air earlier tonight due to the failure of a UPS.
The power has been bypassed to unfiltered; this could mean another sudden crash of the testbed; we hope not. Many of the machines came back up on their own so rather than take them back down again, we completed the restart.
If you can delay your work for a couple of days, ti might be prudent to do so
|Problems swapping in|
|Tuesday, March 3, 2015, 1:39PM; posted by jelena.|
We are currently experiencing some problems swapping in experiments and are working to resolve them.
The cause has not yet been determined, although it seems to come and go and the testbed is currently functioning.
|Metasploitable2 Image Available|
|Tuesday, September 2, 2014, 5:00PM; posted by bks.|
"an intentionally vulnerable version of Ubuntu Linux designed for testing security tools and demonstrating common vulnerabilities."
|Upgrading ports during todays downtime.|
|Wednesday, November 6, 2013, 6:19PM; posted by jhickey.|
I'm using the standard scheduled downtime today to freshen up the ports on boss and users. Swaps should be enabled by 7PM Pacific.
|node_list moved to node_summary, node_list restored and make consistent|
|Friday, October 11, 2013, 5:50PM; posted by faber.|
The node_list command now queries the XMLRPC interface as it used to do. Users who have been using script_wrapper.py node_list to access this function should use /usr/testbed/bin/node_list instead.
The node_list -c flag that outputs containerized node names has been modified in two ways. First, names are produced without the DNS qualifiers as node names provided by other options of this commmand are. A node in a VM container named a will be reported as a not a.exp.proj as earlier versions of this feature did.
The node_list -c flag now reports embedded_pnode containers as well (physical machines). If no containers VMs are present in an experiment, node_list -c and node_list -v produce identical output.
The node_list command is now available as node_summary. It is otherwise unchanged.
|Containers now report hostname in same format as physical nodes|
|Friday, October 11, 2013, 11:25AM; posted by faber.|
Users of containers should be aware that both openvz and qemu containers now report hostname as a fully-qualified domain name (FQDN) just as physical hosts do.
A container named a in the tcl file will now report its name as a.exp.proj.isi.deterlab.net rather than a.
This should simplify moving from physical nodes to container VMs.
|Kali 1 Replaces Backtrack 5R3|
|Monday, August 12, 2013, 12:03PM; posted by bks.|
This year Offensive Security named their penetration testing release "Kali" instead of "Backtrack." DETER now supports Kali 1 instead of last year's Backtrack 5R3.
|Changes to DETER instructional policies and proceedures, 2013|
|Friday, July 12, 2013, 2:47PM; posted by sklower.|
1.) We will no longer generate passwords in advance for new accounts; when students are assigned to the class they will get an email with the URL for "forgot your password", prepopulated with their class ID, as part of the welcoming letter.
2.) On the day that the end-of class is reached, all accounts will be locked - encrypted password set to '*', ssh keys and ssl certs deleted, web_frozen asserted; and email will be sent to the instructor appraising them of the action.
3.) 4 weeks after the class end date, all accounts will be *wiped* (files removed, email aliases removed, new ssl certs, generated).
2 weeks after the class end date, we will send email to the instructor reminding them of the imminent wipeage. If the instructor wishes to delay this, they can update the end-of-class date.
4.) We will formalize the notion of incompletes; a student account
which is marked as being granted an incomplete will not be wiped.
5.) Class limits will be strictly enforced for normal students;
there will be an entry in the group_polices table setting the
limit for nodes to be 0 whenever the instructor has not input
a schedule entry.
6.) Instructors will be unable to assign students to classes until a future end date is set.
We believe the only active class currently using DETER is the ComSecHu project - all others have been assigned a presumptive end date of July 16th, so class accounts will locked on that day, and wiped 4 weeks later.
|Regenerating testbed-specific keys this Wednesday Evening|
|Saturday, January 19, 2013, 5:59PM; posted by sklower.|
Clarification:The account was not compromised at DETER itself. This change was based on our experience with an intrusion outside of DETER where the collection of ssh private keys was part of the attackers toolkit.
Due to a compromised account, we believe that it is prudent to configure ssh in such a way that when one logs into users or boss from the outside the sshd will consult an authorized_keys file which is not exported to experimental nodes and would not contain public keys generated on your behalf by the testbed.
A file users:/etc/ssh/external_keys/
In order to log into experimental nodes, and from one testbed node to another, locally generated ssh public and private keys will be where they used to be.
Any public keys that you have previously uploaded via the web interface into the testbed should continue to work, and both places should be automatically maintained on your behalf.
We were, of course unable to run the utility to make the transition during the UN-scheduled downtime on saturday morning, and will do instead on wednesday.
|Upgrading Serial Controllers|
|Friday, December 21, 2012, 4:49PM; posted by jhickey.|
Serial console access will be interrupted for the pc3060 machines and the pc3000 machines this evening.
|Heavy testbed use until 9/20|
|Thursday, September 13, 2012, 11:38AM; posted by sunshine.|
We have two big research events coming up - one next week and one in October. Users may see shortage of nodes at least until 9/20 as research groups prepare demos. We expect availability of nodes to improve after 9/20. Please let us know if this creates problems for you and we'll try to help you out.
|Testbed Downtime 6PM until 6AM Pacific Time this Thursday|
|Tuesday, August 28, 2012, 3:54PM; posted by jhickey.|
We will be hosting a very large experiment during this time. Access to testbed nodes will be restricted and all experiments will be swapped out in order to accommodate the large experiment.
|Special Downtime Sunday, August 26 at 12PM PST|
|Saturday, August 25, 2012, 6:00PM; posted by jhickey.|
I need to reboot our main server that controls the www.isi.deterlab.net web interface and controls experimental swap-ins. This downtime should only last ~15min.
|Large experiment this Thursday|
|Sunday, August 19, 2012, 1:16PM; posted by jhickey.|
We will be swapping in a very large experiment this Thursday. Testbed usage will be restricted and active experiments may be swapped out.
|Ubuntu1204-64-STD image released|
|Wednesday, July 25, 2012, 2:14PM; posted by bks.|
Enjoy; please file a ticket for any issues.
|Experimental Gateway Fixed|
|Tuesday, July 24, 2012, 2:59PM; posted by jhickey.|
We ran into hardware problems with our old Experimental gateway that bridges the ISI part of the testbed with the UCB part of the testbed. I had some new hardware already mostly setup to replace the old machine, so I deployed it this afternoon to get things connected again.
|Special Downtime Tuesday, July 17 at 8PM Pacfic|
|Monday, July 16, 2012, 6:35PM; posted by jhickey.|
I will be upgrading the ISI side of the link between ISI and UCB. Additionally I will be moving the serial server responsible for power control for the pc2133 and pc3000 machines. I expect the downtime to last about an hour.
|Swaps disabled (enabled now)|
|Wednesday, June 13, 2012, 9:00PM; posted by jhickey.|
I am working on fixing the event system. Swaps will be enabled when things are fixed.
Update: The event system should be fixed now.
|Testbed back in order|
|Sunday, May 27, 2012, 2:01PM; posted by jhickey.|
The event system server died on users last night. This was preventing experiments from fully swapping in. Everything should be back up and running now.
|Quotas enabled for project/group directories|
|Thursday, May 17, 2012, 4:49PM; posted by jhickey.|
/proj is getting full, so it is time to enable quotas. We have given all projects a soft limit of 10Gb and a hard limit of 30Gb. Please contact us if you have any questions or need more space.
|CSET '12 submission EXTENDED to April 26th|
|Friday, April 20, 2012, 11:12AM; posted by sunshine.|
Please consider submitting a paper on your experimental research to CSET '12, held in conjunction with the USENIX Security Symposium, Aug. 6 in Bellevue, WA. Visit http://www.usenix.org/events/cset12/cfp for details. Deadline is EXTENDED to April 26th.
|PBS NewsHour Story on DETER|
|Wednesday, April 18, 2012, 2:15PM; posted by jhickey.|
The PBS NewsHour aired a story about DETER on April 16. Watch the segment at:
|Downtime tonight at 10PM Pacific for database work|
|Tuesday, March 27, 2012, 4:04PM; posted by jhickey.|
I will be doing some work on the database tonight. Experiment swapping as well as the web interface will be unavailable during this time. I expect the downtime to last less than an hour.
|Windows XP Image updated|
|Wednesday, March 21, 2012, 6:13PM; posted by jhickey.|
The standard WINXP-UPDATE image has been updated to include all recent security patches. Additionally, we now include .Net support by default (Version 3.5 and 4.0 are installed).
|Ubuntu1004-STD image updated|
|Friday, March 2, 2012, 1:27PM; posted by jhickey.|
We fixed an issue where Ubuntu1004-STD would sometimes fail to boot properly. Swap in reliability for large experiments should be improved. For instructions on updating your custom images made before March 2, 2012, please refer to the Ubuntu wiki page.
|Experiment Isolation being reinstated|
|Friday, February 17, 2012, 5:31PM; posted by sklower.|
We will resume experiment isolation, turning it off and on a couple of times during the scheduled downtime tomorrow (sat 2/18).
With ~30 experiments, swaps should be frozen for about 3 and a half minutes each time, with a period of disconnection for each node being less than 30 seconds.
|Berkeley Switch Interconnect Upgraded to 28 - 40 Gbs|
|Wednesday, February 1, 2012, 7:40PM; posted by bks.|
This is more than an order of magnitude improvement. Also, all Berkeley switches are HP Procurves.
|Brief downtime tonight at 8PM Pacific: Bad memory in Boss.|
|Saturday, January 21, 2012, 7:03PM; posted by jhickey.|
Boss is reporting that one of the memory DIMMs is failing. I will be shutting down boss tonight to swap the module out. I will also be using the downtime to upgrade the web server on boss.
|ISI now using 20Gbe star topology and Nortel10 replaced.|
|Saturday, January 14, 2012, 2:34AM; posted by jhickey.|
All experimental switches at ISI are now connected to a single HP 5412 switch using a pair of 10Gbe interfaces for each of the 6 experimental 5412 switches. This is a major upgrade over our previous experimental topology at ISI which was only partially 10Gbe. Also, the last Nortel switch which was serving the pc3000 machines has been replaced with a pair of HP 5412 switches. The ISI side of the switch upgrade is now complete.
|Experimental Switch for the pc3000 machines to be replaced Friday Afternoon|
|Wednesday, January 11, 2012, 11:06PM; posted by jhickey.|
As the final part of the ISI switch upgrade, I will be replacing the Nortel stack that currently supports the experimental interfaces of the pc3000 machines. Additionally, I will be switching our network topology to a star topology with 20 Gbit interconnects. Experimental swap-ins will be limited during this time and experiments that incorporate multiple node types at ISI (this switch currently acts as the head switch at ISI) will have their connectivity disrupted. I should begin work on this around 3 PM Pacific time. Swapping out such large switch stacks takes a fair bit of time. I expect the testbed to be back on the air by 12 AM Friday night.
|Experimental switch for pc2133 and NetFPGA machines to be replaced on Monday.|
|Friday, December 16, 2011, 2:58PM; posted by jhickey.|
I will be replacing the Cisco 6509 that is currently functioning as the control network switch for the pc2133 and NetFPGA machines Monday afternoon. The nodes will be unavailable during the upgrade and any experiments using them will be swapped out.
|Enhanced Network Delay/Loss functions for DETER|
|Wednesday, December 14, 2011, 2:39PM; posted by faber.|
DETER announces the deployment of the new delay nodes, based on the Click modular router. Beside supporting functions provided by old delay nodes, they facilitate specifying sophisticated loss models that allow burst losses. Moreover, they emulate variable and static delays. Variation in delay is described by Normal, Poisson or Exponential distribution. We encourage users to use them. All you have to do is to replace “make-lan” or “duplex-link” by “make-deter-lan”. For more information please visit. https://trac.deterlab.net/wiki/NewDelayNodeNs
|Experimental Switch for the pc3060 and pc3100 machines upgraded.|
|Friday, December 2, 2011, 11:37PM; posted by jhickey.|
We replaced our aging Nortel experimental stack that serves the pc3060 and pc3100 machines with a pair of new HP 5412zl switches tonight. We will be upgrading the reset of the experimental network here at ISI over the next few weeks.
|pc3060 nodes unavailable on Friday, December 1.|
|Monday, November 28, 2011, 6:44PM; posted by jhickey.|
I will be replacing the experimental switch currently serving the pc3060 and pc3100 machines here at the ISI side of DETER. I will start removing these machines from the available pool of nodes starting on Thursday. The switch upgrade should take a few hours and the machines should be back in service by Friday evening. Experiments still using these nodes after 12PM on Friday will be swapped out.
|Limited availability of DETERlab experimental nodes, November 11 - 21, 2011|
|Thursday, October 27, 2011, 3:53PM; posted by jhickey.|
During the period November 11-21 (and especially Nov 12-18), an important research program will have priority use of the DETERlab testbed. During this period, users may experience difficulty swapping in large experiments, and at times all users may be preempted by forced swap out. If the nodes you need are available, you can use them, but understand that your nodes may be preempted with little advanced warning. Refer questions to email@example.com.
|Testbed updated (New OS, Packages, and updated Testbed Software)|
|Sunday, September 18, 2011, 8:07AM; posted by jhickey.|
Boss and users have been updated to FreeBSD 8.2 with current packages. The package build took much longer than expected. The testbed software has also been updated to the latest in our repo. If you suspect a bug, please do not hesitate to let contact us.
|Main testbed servers getting upgraded tonight around 10 PM Pacific Time (Saturday)|
|Saturday, September 17, 2011, 5:02PM; posted by jhickey.|
I will be upgrading boss and users late tonight to minimize disruption. The process should take a few hours since I will be upgrading the packages on these hosts (in FreeBSD packages are compiled from source).
|New OS Images: Click Modular Router (Ubuntu804-click and Ubuntu1004-click20)|
|Thursday, September 15, 2011, 2:31PM; posted by mikeryan.|
We are happy to announce official support for the Click modular router on DETER. Please use Ubuntu804-click or Ubuntu1004-click20
|BackTrack 5 R1 Image Available|
|Tuesday, September 6, 2011, 3:21PM; posted by bks.|
Last month's release of the BackTrack penetration testing arsenal is available as BT5R1. To use access the node from your desktop, log into the node and start a VNC server:
pcNNN% sudo -H vncserver -geometry 1200x900then start a viewer on your workstation:
workstation% vncviewer -via users.isi pcNNN:1This works for tightvnc; you could also start your own ssh tunnel.
|FBSD8-STD updated to FreeBSD 8.2|
|Tuesday, July 19, 2011, 1:04PM; posted by bks.|
We updated FBSD8-STD from FreeBSD 8.1 to 8.2. If for some reason your experiment requires 8.1, that remains available as FBSD81-STD.
|CentOS5 image updated to CentOS 5.6|
|Monday, July 18, 2011, 4:33PM; posted by jhickey.|
We have mirrored CentOS 5.6 to our local package mirror and updated our CentOS5 image to the latest 5.6. Please file a ticket if you run into any problems.
|Serial console restored for remaining PCs.|
|Saturday, June 25, 2011, 11:39PM; posted by jhickey.|
pc001 through pc064 should all have serial console now.
|Serial consoles restored for pc001 through pc031. pc032 through pc064 will be restored tomorrow.|
|Friday, June 24, 2011, 7:33PM; posted by jhickey.|
We replaced the failed serial server on Thursday evening, but we are being slowed down because the pin-outs for the RJ45 connectors on the new server are different from the old one. Luckily, we can rearrange the pins on our existing inventory of adapters, but it takes a little time to do.
|Serial server for the pc2133 machines down|
|Thursday, June 23, 2011, 2:25PM; posted by jhickey.|
We are experiencing problems with the serial server that serves the pc2133 machines. I am in the process of replacing it, since we were intending to do so anyway. This will take a little bit of time. I expect serial consoles to be back up later in the day.
|Introducing ThirdEye is a semantic analysis framework|
|Tuesday, June 21, 2011, 2:22AM; posted by jhickey.|
We have a new analysis tool for researchers to use with DETERlab experiments: ThirdEye. ThirdEye is a semantic analysis framework to identify interesting relationships in network and cybersecurity experiment data. If you want to explore what is going on in your experiment, or automatically ensure every experiment trial is valid, you will find ThirdEye useful.
Read more about ThirdEye at http://thirdeye.deterlab.net
|Rebooting boss and users tonight (Wed, June 15) at 10:00pm Pacific time|
|Wednesday, June 15, 2011, 7:04PM; posted by jhickey.|
We will be preforming a quick reboot of both boss (www.isi.deterlab.net) and users. Experimental swaps will be disabled during this time.
|Bug in New Project form fixed|
|Tuesday, May 17, 2011, 9:23AM; posted by mikeryan.|
If you attempted to create a new project in the last two days and there was a failure, please resubmit your project application. We apologize for the inconvenience.
|service outage this morning has been resolved|
|Friday, May 6, 2011, 9:25AM; posted by mikeryan.|
We had an internal service outage this morning which resulted in:
We've tracked it down and fixed it. Connectivity and node availability should be restored to normal.
|DETER Chat (IRC) no longer official support medium|
|Thursday, April 14, 2011, 4:16PM; posted by mikeryan.|
We are no longer supporting DETER Chat (IRC) as an official support medium. Please follow instructions on the contact page if you have an issue you need to report.
|More bpc2133 problems ... cooling is failing|
|Friday, April 1, 2011, 4:27PM; posted by sklower.|
We may have to power down the bpc2133 nodes quite soon will post again when it is resolved.
|Switch Firmware Update Will Disrupt UCB Node Connectivity Friday Afternoon|
|Thursday, March 24, 2011, 5:11PM; posted by bks.|
We hope the disruption will be short, but in the worst case connectivity will be unstable between 12:00 and 17:00 PDT.
|Web Login by email address is no longer supported.|
|Wednesday, March 23, 2011, 6:01PM; posted by jhickey.|
DETER no longer supports using your email address instead of your DETER username when logging into the web interface. Allowing the use of email addresses instead of usernames for the web interfaces appears to have caused a fair amount of confusion for new DETER users as to what their username really is. In order to be consistent with the username supplied when logging into testbed nodes and users, we have disabled the ability to use an email address for the web interface.
|Rebooting the internet facing firewall at ISI today at 5PM Pacific Time.|
|Monday, March 7, 2011, 2:23PM; posted by jhickey.|
We will be rebooting the ISI internet facing firewall today at 5PM to complete an update. DETER will be unreachable for a few minutes as the machine boots.
|Wednesday, March 2, 2011, 5:19PM; posted by bks.|
John Hickey fixed the connection between a couple switches at ISI.
|Disconnection Anomaly Noted Between DETER Campuses|
|Tuesday, March 1, 2011, 6:30PM; posted by bks.|
Deter Ops have noted an instance of unexpected disconnection between nodes at Berkeley and ISI, ie between names bpcNNN and pcMMM, on experimental interfaces. We do not know how widespread the problem is, but are working on it. bpcNNN can reach pcMMM where MMM <= 60, so the disconnection appears to be between two ISI switches.
|Additional Regular Downtimes: Saturday Mornings 10am-1pm Pacific|
|Friday, February 25, 2011, 1:02AM; posted by sklower.|
We will be conducting control-net separation related testing on DETER saturday mornings, on a regular basis.
This coming saturday (2/26) we anticipate a period of about 40 minutes in which experiments may not be swapped in or out; depending on how well things go, we may leave control-net separation in effect.
Nodes within an experiment will be able to comunicate with each other, the boss and users nodes over the control net, but will not be able to send or receive traffic from nodes in other experiments.
|Wednesday, February 16, 2011, 12:09AM; posted by jhickey.|
I have applied the latest patches and integrated a tweak or two into the standard Ubuntu 1004 image.
|bpc2133's are available again|
|Thursday, February 10, 2011, 6:56PM; posted by mikeryan.|
Power to the black box that houses the bpc2133's has been restored and the nodes are available once again.
|CSET'11 Call for Papers|
|Monday, February 7, 2011, 12:05PM; posted by sunshine.|
Please consider submitting your paper to CSET'11 conference, held in conjunction with USENIX Security Symposium, August 8, 2011 in San Francisco, CA. This year, we're accepting both regular papers, position papers and extended abstracts. Submission deadline is April 18. For more information please visit: http://www.usenix.org/event/cset11/cfp/.
|Downtime extended until 7PM|
|Saturday, February 5, 2011, 6:10PM; posted by jhickey.|
It will be about another hour before things are operational again. Sorry for the delay.
|Additional Testbed downtime TODAY until 5PM Pacific Time|
|Thursday, February 3, 2011, 3:44PM; posted by sklower.|
We will be replacing the control net switches between 10:30am and 5pm on Saturday February 5th.
Nodes in your long-running experiments may be unreachable for periods up to 3 hours at a time.
|Users upgraded to FreeBSD 7.4-RC3|
|Tuesday, February 1, 2011, 2:08PM; posted by jjh.|
In order to solve some NFS locking problems, we have upgraded to FreeBSD 7.4-RC3. So far, things seem to be working fine.
|Rebooting USERS today, Feb 1, at 2PM Pacific Time.|
|Tuesday, February 1, 2011, 12:10PM; posted by jjh.|
We are upgrading to the release candidate of FreeBSD 7.4 in order to fix some NFS locking problems we have been experiencing. This should be a quick downtime of about 15 min.
|UC Berkeley connectivity restored, bpc2133's remain unavailable|
|Thursday, January 27, 2011, 1:47PM; posted by mikeryan.|
Our connection at UC Berkeley was dropped this morning, and a backup link has been brought up. The facilities housing the bpc2133's remains unpowered and those nodes are still unavailable.
|We are experiencing problems with the UCB side of the testbed.|
|Thursday, January 27, 2011, 9:26AM; posted by jjh.|
We are working on fixing the issue. Please stand by.
|bpc2133 Nodes Unavailable|
|Thursday, January 27, 2011, 9:01AM; posted by bks.|
The Sun Mobile Data Center which houses the bpc2133 nodes shut itself down.
|CentOS5 Image Updated with NetFPGA support.|
|Thursday, January 20, 2011, 4:16PM; posted by jhickey.|
We now have built-in support for NetFPGA devices in our standard CentOS5 image. Additionally, all recent updates have been applied to the image and we are now tracking RPMforge for packages that are not included in the CentOS distribution.
|Ubuntu1004-STD image updated.|
|Tuesday, January 4, 2011, 3:31PM; posted by jhickey.|
We have applied the latest updates to the image and added in a new software repository. Please let us know if you run into any issues with the new image.
|Taking disk images and snapshots made easier.|
|Thursday, December 2, 2010, 7:12PM; posted by jhickey.|
We have changed the way that disk images and snapshots are created. We now have links in the 'Reserved Nodes' list in Experiment Information to easily allow you to create new disk images and take snapshots.
By specifying the node up front, we are able to inherit all the metadata from the image that is currently running on the node. This means that end users will no longer need to select partitions or which nodes an image supports. We hope this will eliminate confusion about which nodes an image supports and what partitions scheme is used.
Please contact us if you run into problems with this updated functionality.
|Problem with pc2133 machines resolved....|
|Thursday, November 11, 2010, 6:01PM; posted by jhickey.|
The experimental switch for the pc2133 machines has been reconnected with the rest of the testbed experimental switches.
|pc2133 machines experiencing problems...|
|Thursday, November 11, 2010, 3:11PM; posted by jhickey.|
We are looking into a problem with the switch that serves the experimental connections for the pc2133 machines.
|Event system restarted...|
|Tuesday, November 9, 2010, 1:35PM; posted by jhickey.|
The event system was stuck. We have restarted the service and are looking into the cause. Experiment swap-ins should work now.
|Berkeley Nodes back to normal|
|Friday, November 5, 2010, 6:26PM; posted by jhickey.|
The boot problems with the Berkeley nodes have been fixed. The problem was related to the testing of control network separation during yesterdays downtime.
|Berkeley nodes experiencing boot problems.|
|Friday, November 5, 2010, 3:52PM; posted by jhickey.|
There appears to be a problem with the testbed nodes at Berkeley causing them to fail to boot from the network. We are still tracking down why they can't boot.
|Tracking down network issue with our connection to the world.|
|Tuesday, November 2, 2010, 5:42PM; posted by jhickey.|
Our link to the outside world is currently running at 100Mbit/Half-Duplex. We are talking with our network provider to resolve this issue.
|Central Nortel control network switch replaced with a HP switch.|
|Monday, November 1, 2010, 6:54PM; posted by jjh.|
In order to better diagnose multicast problems that we have been having with control network separation, we have replaced one of the main Nortel control network switches with a HP 2810 switch.
|Disk Quotas applied to user accounts.|
|Friday, October 29, 2010, 2:27PM; posted by jhickey.|
A quota of 10GB with a temporary maximum of 20GB has been set for all users. Please file a ticket if you require more space in your home directory.
|Special Downtime, Monday November 1st from 6PM until 8PM.|
|Friday, October 29, 2010, 1:57PM; posted by jhickey.|
I will be swapping out one of our Nortel control network switches with a HP model. This is the main control network switch, so the entire control network will be interrupted during this downtime. The actual swap should be fairly quick, but I am scheduling two hours in case something goes wrong.
|Rebooting users again.|
|Thursday, October 28, 2010, 9:11PM; posted by jhickey.|
Ran into some NFS problems. It will take about 30 min for users to come back on line. Sorry for the unscheduled downtime.
|Users rebooted with new kernel to address disk space issues.|
|Thursday, October 28, 2010, 7:41PM; posted by jhickey.|
Users was rebooted with a new kernel with disk quota support. It is currently coming back online.
|/mnt/other filesystem is full. This will be cleared up shortly.|
|Thursday, October 28, 2010, 6:07PM; posted by jhickey.|
The /mnt/other filesystem is full and we are working with some heavy users to clean up their home directories.
|CentOS 5 image updated.|
|Tuesday, October 26, 2010, 8:08PM; posted by jhickey.|
The same mkextrafs.pl fix that was applied to the Ubuntu1004-STD image has now been applied to the CentOS 5 image.
|Ubuntu 10.04 LTS Image updated|
|Tuesday, October 26, 2010, 7:15PM; posted by jhickey.|
The new image fixes a bug in the mkextrafs.pl script which caused incorrect 3rd and 4th partitions to be created. These partitions in turn caused a problem when creating disk images after mkextrafs was run. Also, the installed packages and kernel were brought up to date.
|Default Operating System changed to Ubuntu1004-STD|
|Friday, October 15, 2010, 6:29PM; posted by jhickey.|
The default operating system for nodes where an operating system has not been specified has been changed to Ubuntu1004-STD from the old FC6-STD image. Please contact us if you run into problems.
|trac.deterlab.net moved to new IP address.|
|Monday, October 11, 2010, 7:30PM; posted by jhickey.|
We have moved trac.deterlab.net to a new IP address (220.127.116.11). It can take up to an hour for the DNS change to propagate.
|Changing trac.deterlab.net's IP address tonight at 7PM Pacific Time|
|Monday, October 11, 2010, 2:50PM; posted by jhickey.|
We will be moving trac.deterlab.net to a different IP address. Trac will be unavailable for about an hour from 7PM until 8PM assuming things go right.
|Updating control network switch firmware during the downtime today.|
|Thursday, September 30, 2010, 3:03PM; posted by jhickey.|
We will be upgrading the firmware of the Nortel switches on the control network today. This will disrupt communication between testbed node and users/boss. We do not expect the downtime to be long, but connectivity may be sporadic as different switches are rebooted.
|Isolation of Berkeley nodes postponed.|
|Wednesday, September 22, 2010, 1:21PM; posted by sklower.|
The Berkeley Campus network staff has cancelled its upgrade of the core router through which berkeley DETER traffic passes. They have not yet posted the revised downtime. Experiments swapped in will continue to function normally between 5am and 7am tomorrow.
|Berkeley blackbox nodes isolated today 4-5:15pm|
|Tuesday, September 21, 2010, 5:30PM; posted by sklower.|
This unplanned outage was the result of an as yet undetermined bug in the control software.
|Berkeley nodes isolated 5am-7am thursday 9/23, swaps may not work|
|Monday, September 20, 2010, 2:52PM; posted by sklower.|
The campus router through which the berkeley side of the testbed is connected to the outside world will be undergoing maintenance from 5am to 7am this coming thursday, 9/23.
Berkeley nodes for swapped in experiments will not be able to access the home directories mounted by NFS, and since the testbed switches will also be unreachable from the ISI boss, swap ins/outs/modifies will fail.
When campus is finishes its maintenance, things will start working again on the testbed.
|Rebooting trac.deterlab.net and other trac project sites at 3pm.|
|Monday, September 20, 2010, 12:55PM; posted by jhickey.|
We will be applying the latest patches which include a new kernel. trac should be unavailable for a few minutes around 3PM pacific time today.
|Updated Ubuntu804-STD and Ubuntu1004-STD|
|Wednesday, September 15, 2010, 6:48PM; posted by jhickey.|
Updated both Ubuntu images to turn off automatic filesystem checks. Also applied any outstanding updates to each image.
|CentOS package mirror now updated automatically.|
|Friday, September 10, 2010, 5:47PM; posted by jhickey.|
The local package mirror (scratch) for CentOS 5.5 is now updated daily. This is in addition to daily updates of the Ubuntu repository.
|Gateway machines updated at ISI...|
|Friday, September 10, 2010, 5:45PM; posted by jhickey.|
The gateway machines that link the ISI control and experimental networks with UCB have been updated to FreeBSD 7.3.
|Additional Downtime Sept 5, 5-7pm|
|Saturday, September 4, 2010, 1:26PM; posted by sklower.|
There will be some more control net separation testing, sunday september 5th from 5pm to 7pm.
Users will not be able to swap in, swap out or modify their experiments during this time, but swapped-in experiments should continue to run, with only about a maximum of 30 seconds of disconnectivity from the testbed at the beginning and end of the period.
It is questionable whether users will be able to reload nodes for swapped-in experiments.
|Modifying swapped in experiments works again|
|Thursday, September 2, 2010, 2:54PM; posted by mikeryan.|
We pulled in a fix from upstream, and all appears to be well.
|CentOS 5 image updated|
|Tuesday, August 31, 2010, 2:38PM; posted by jhickey.|
We have updated the CentOS 5 image (CentOS5) to CentOS 5.5.
|New experiments not swapped in by default|
|Thursday, August 26, 2010, 11:53AM; posted by mikeryan.|
When you create a new experiment the default behavior is to create the experiment but leave it swapped out. If you want to swap it in as soon as it's created, check the 'Swap In Immediately' box.
|Ubuntu 10.04 LTS Image updated|
|Friday, August 20, 2010, 7:43PM; posted by jhickey.|
The Ubuntu1004-STD image has been updated to tweak how partitions work (switched from UUID to labels in /etc/fstab) and all recent security updates have been applied.
|Using DETER to teach a class this fall?|
|Thursday, August 12, 2010, 2:01PM; posted by jhickey.|
If so, please send a heads up to firstname.lastname@example.org. We need to know you deadlines, TA contacts, and expected class sizes.
|Brief downtime this Friday (July 23) from 5PM until 5:30PM PST|
|Thursday, July 22, 2010, 5:36PM; posted by jhickey.|
I will be patching the VMWare server that runs trac, irc, and one of the UCB serial servers. I do not expect this to take more than 30 min.
|Downtime extended to 7PM|
|Tuesday, July 20, 2010, 6:31PM; posted by jhickey.|
We are extending the planned downtime for today until 7PM. Sorry for any inconvenience.
|Special Downtime from 5PM until 6PM PST on Tuesday, July 20|
|Thursday, July 15, 2010, 1:47PM; posted by jhickey.|
We will be upgrading the switch firmware on the Nortel switches to the latest release which should fix some multicast issues introduced with the previous release. This will affect the control network since the switches need to reboot when the firmware is upgraded. You can track the completion of the upgrade by checking this ticket in trac: https://trac.deterlab.net/ticket/124 (login required). We will also be rebooting boss and users during this time.
|CSET 2010 Workshop on August 9th in Washington DC|
|Tuesday, July 13, 2010, 9:43PM; posted by sridhar.|
You are invited to participate in the 3rd CSET (Cyber Security Experimentation and Test) 2010 workshop being held in Washington DC on Monday, August 9th. The Early Bird Registration Deadline is Monday, July 19, 2010 to receive the greatest savings.
Registration and details of the workshop are available at: http://www.usenix.org/events/cset10/index.html
Attention, students! CSET '10 has a limited number of student travel grants available. The deadline for applying is July 15.
CSET focuses on the science, design, architecture, construction, operation, and use of cybersecurity experiments in network testbeds and infrastructures. The workshop's scope includes all work relevant to cyber security experimentation and evaluation including simulation, emulation, deployment, traffic models.
We have an interesting and stimulating program this year, including a keynote address by Dr. Doug Maughan, Program Manager, U.S. Department of Homeland Security's Science and Technology Directorate on "The Role of Testbeds in CyberSecurity Research" along with presentations on Cyber Physical systems, emulation testbeds as well as key sessions on Security Education, Work in Progress and brainstorming sessions. In addition, a discussion on "Security Experimentation with Cyber-Physical Devices" is scheduled with a panel that comprises of key practitioners in the field.
We look forward to seeing you in Washington DC!
|Modifying swapped in experiments will not work properly|
|Thursday, July 8, 2010, 1:12PM; posted by jhickey.|
DO NOT MODIFY SWAPPED IN EXPERIMENTS
If you want to modify your experiment, swap it out first.
We are working on fixing this ASAP. We apologize for any inconvenience.
|FreeBSD 7 Image updated to support local package mirror.|
|Monday, July 5, 2010, 11:04PM; posted by jhickey.|
We have placed the binary packages released with FreeBSD 7.3 on scratch and have updated root's .cshrc on the FBSD7-STD image to automatically download from scratch. To install precompiled binary packages on FreeBSD 7, simply use the -r switch with pkg_add.
|Ubuntu 10.04 LTS Image released|
|Tuesday, June 29, 2010, 4:03PM; posted by jhickey.|
A new Ubuntu 10.04 LTS image is now available. The osid is Ubuntu1004-LTS. Please file a trouble ticket if you run into problems with the image.
|Upgrading users and boss to FreeBSD 7.3 during the Thursday downtime.|
|Monday, June 21, 2010, 9:54PM; posted by jhickey.|
FreeBSD 7.2 is going end of life at the end of the month. I will be upgrading users and boss to FreeBSD 7.3. Both machines will be rebooted during the usual Thursday downtime.
|Berkeley bpc2133 nodes once again available|
|Thursday, June 17, 2010, 7:40PM; posted by sklower.|
the cooling system in the berkeley black box has been repaired. The failure was due to a mixing valve under servo control; the water supplied by the building is too cold, and so it it is mixed with recirculated water. The valve at this location seems to fail and need to be replaced every 6 - 9 months; we now keep a replacement part *on site*, but we are not a liberty to replace it ourselves; only somebody from the central campus maintenance division is authorized to inspect, and either replace themselves or hire a contractor, and the office responsible is not staffed on weekends. At least this time we did not have to wait an extra 3 days to have a spare valve manufactured and shipped to us, but it is almost certain that this will happen again in another 6 to 9 months.
|DETER Project Review|
|Wednesday, June 9, 2010, 8:35AM; posted by jhickey.|
We are having our project review on June 9th and 10th. Any review related demo experiments will be given priority on the testbed. We do not expect interruption of service as we have already allocated the necessary resources to our experiments.
|Downgrading Nortel10 firmware today at 7PM PST|
|Friday, May 28, 2010, 5:04PM; posted by jhickey.|
The control network will be interrupted for about 20 min while I downgrade the firmware on Nortel10 to work around a multicast issue that seems to have been introduced with the latest Nortel firmware.
|Extended Control Net testing this week|
|Sunday, May 23, 2010, 11:11PM; posted by sklower.|
Control Net Separation Testing will be happening Monday and tuesday between 3 and 6pm, Wednesday, thursday and friday between 5 and 8 pm
|Updated FBSD7-STD image|
|Tuesday, April 27, 2010, 10:27PM; posted by jhickey.|
We have updated the FBSD7-STD image to FreeBSD 7.3. This updated image features larger partitions to make building custom kernels easier and the source for FreeBSD 7.3 is in /share/freebsd/7.3.
|Updated CentOS 5 image|
|Tuesday, April 27, 2010, 3:23PM; posted by jhickey.|
The CentOS image has been updated in order to fix a compatibility problem with SEER and to install the latest security fixes.
|Special downtime Tuesday, May 4th between 1PM and 3PM PST|
|Tuesday, April 27, 2010, 2:29PM; posted by jhickey.|
We will be enabling control network separation on Tuesday May 4th. There should be about a 15 minute interruption on the control network between 1PM and 3PM PST.
|Student travel grants for IEEE Security and Privacy Symposium|
|Wednesday, March 31, 2010, 12:48PM; posted by sunshine.|
There's a significant amount of student travel grants available for IEEE Symposium on Security and Privacy in Oakland, California. This is a premier conference in security and privacy. The eligibility criteria is that one must be a student at a US institution.
For more information and to apply see: http://oakland31.cs.virginia.edu/grants.html. The deadline is April 2 but it may be extended.
|Rebooting main DETER firewall during the downtime on Thursday.|
|Wednesday, March 31, 2010, 1:26AM; posted by jhickey.|
The main firewall for DETER will be updated to FreeBSD 7.3 and rebooted during our normal downtime on Thursday, April 1. The downtime should be minimal.
|CFP: CSET 2010 (CyberSecurity Experimentation and Test Workshop)|
|Monday, March 22, 2010, 12:22PM; posted by sunshine.|
On behalf of the 3rd Workshop on Cyber Security Experimentation and Test (CSET '10) program committee, we'd like to invite you to submit papers on the science, design, architecture, construction, operation, and use of cyber security experiments in network testbeds and infrastructures. Please submit all papers by May 24, 2010, 11:59 p.m. PDT.
Topics of interest include but are not limited to:
|CFP: International Symposium on ICT System Testbeds|
|Monday, March 15, 2010, 2:30PM; posted by sunshine.|
Developments in cloud computing and ubiquitous network computing have increased the reliability and safety of advanced large-scale network systems, and driven the demand for rapid advances in these systems. Enhancing and upgrading the testbeds used to test network systems has become a necessity.
National Institute of Information and Communications Technology (NICT) and Japan Advanced Institute of Science and Technology (JAIST) will hold "International Symposium on ICT System Testbeds" on March 30, 2010.
This symposium includes lectures by specialists from around the world on international trends and research findings in ICT system testbeds, as well as future prospects and expectations, and examines the future direction for ICT system testbeds in Japan.
For detailed information of the symposium program and participation, please visit http://starbed.nict.go.jp/ictstb.htm We hope you will attend this symposium.
|LLNL Student Internship Program|
|Thursday, March 11, 2010, 2:44PM; posted by sunshine.|
LLNL is looking for summer interns for their Cyber Defender Program. They are particularly interested in people that had DETERlab experience. For more information and to apply please visit https://cyberdefender.llnl.gov/.
|Special downtime from 5PM until 7PM PST on Tuesday, March 9.|
|Monday, March 8, 2010, 4:05PM; posted by jhickey.|
We ran into problems with multicast after upgrading the firmware on the Nortel control net switches last Thursday. We were able to work around the problem by enabling IGMP snooping, but snooping may be problematic when used in combination with control network separation. We will be looking further into the multicast issue during this downtime.
|Rebooting users today|
|Monday, March 8, 2010, 3:58PM; posted by jhickey.|
In order to address some performance issues I have recompiled the kernel on users to take out some extra debugging features. Users will be rebooted at 6PM PST today.
|Nortel Firmware updated...|
|Thursday, March 4, 2010, 7:30PM; posted by jhickey.|
We have updated all Nortel switches in the testbed to the latest firmware available which is supposed to fix the vlan creation problem.
|UCB to ISI link performance problem fixed.|
|Thursday, February 25, 2010, 11:11PM; posted by jhickey.|
The control net interface on the ISI gateway was set to down 100mbit after replacing Foundry10 causing periodic performance problems for the UCB nodes (in particular whenever nodes were reloading). We have reconfigured the interface back to 1000mbits and things seem to be working better now.
|Foundry4 control net switch replaced|
|Thursday, February 25, 2010, 11:09PM; posted by jhickey.|
Tonight during the downtime (and two hours beyond the downtime) we replaced the remaining Foundry switch on our control network with a pair of Nortel gigabit switches. We apologize that the downtime extended beyond the normal 2 hour window.
|Replacing Foundry4 control network switch this evening.|
|Thursday, February 25, 2010, 5:21PM; posted by jhickey.|
During today's normal downtime, we will be replacing our last Foundry control network switch with a pair of Nortel switches. The downtime may last a little longer than usual, but we expect the testbed to be operational again by 9PM PST.
|Rebooting Nortel10 and Nortel18 experimental switches|
|Tuesday, February 23, 2010, 7:37PM; posted by jhickey.|
We are having issues with these switches and will be rebooting them shortly (at 8PM PST). Sorry for any inconvenience.
|NetFPGA machine integrated into the testbed.|
|Friday, February 12, 2010, 5:45PM; posted by jhickey.|
We have added a machine with a NetFPGA board into the testbed. It will be primarily used to support some classes, but feel free to contact testbed-ops if you are interested in using the machine when it is free.
|Kernel bug fixed|
|Monday, November 30, 2009, 10:56AM; posted by jhickey.|
Last week we fixed the kernel memory leak that was causing users to run out of kernel memory about every two days. If you are interested in the details, you can view the FreeBSD problem report w/patch: kern/140853: [nfs] [patch] NFSv2 remove calls fail to send error replies (memory leak!)
|Problems loading WINXP-UPDATE should be fixed.|
|Monday, November 2, 2009, 6:54PM; posted by jhickey.|
An obscure testbed software bug has been preventing WINXP-UPDATE from loading has been found and fixed for the pc3000 machines. Sorry for any inconvenience.