News - Nov 2003

Nov 27 2002 - The ACCC Help and Answer Database (AHEAD, a.k.a. consult@uic.edu) will be unavailable on Thursday, November 28, 2002. Any mail sent to the database will be received, but not processed, until late Thursday or early Friday. We apologize for the inconvenience. -AHEAD Maintenance

Nov 24 2002 - Lost peering with ICN between 9:45 and 10:15 at the same time a massive DOS attack was being launched out of the ITS network. Darlene Senior Operator Computer Center

Nov 23 2002 -

Executive Summary:

We experienced a failure in the file system (storage container) that holds email inboxes on tigger. This caused a corruption problem and it took many hours to reconstruct the inboxes from backups and from the corrupted area. The vast majority of inboxes were recovered completely, however some people may see duplicate copies of messages. Inboxes for a small group of people may be "backleveled" to the way they existed on Thursday night. See the last paragraph of this message if you need assistance.

Details:

At about noon on Friday November 22, tigger crashed due to an apparent software flaw in the "logical volume" driver of the AIX operating system. The problem developed when we were trying to make use of an AIX feature that allows data to be moved around to different disks even while the system is up and running. The intent was to improve performance by moving a very active component of the file system from a busy disk to an idle disk.

We have used this feature many times in the past and have never experienced this kind of failure. We will contact IBM support about the problem and we will refrain from using this migration feature while the system is active.

This unfortunately caused a corruption of the large file system that stores the email "inboxes" on tigger. Recovery from this problem took more than 12 hours, due in large part to the huge amount of data involved (11,000 inboxes).

We decided to rebuild the file system as best we could from backups taken the night before, merged with what we could save from the corrupted area (which luckily was quite significant). We felt it was worth the extra down-time to restore the inboxes to the most recent state possible.

The result is that a small number of inboxes may have been backdated to the way they appeared Thursday night. The vast majority were restored to the state they were in at the moment of failure. However, it is possible that you may experience some resurrections of emails that you thought you had viewed and/or deleted. This problem will affect pine, imap/webmail and pop users differently.

We are continuing to monitor the situation all weekend.

If you have too much email (i.e. resurrections of deleted items) please just delete again. However, if you think you are missing mail items, or if your inbox cannot be opened, email systems@uic.edu or telephone Network Operations (24/7) at 312-413-8080 ext 2.

Nov 14 2002 - This morning's mailserv problem was precipitated by a triggered circuit breaker. (Physical Plant engineers are here right now measuring the current usage on the questionable circuit and we will reorganize as needed over the weekend).

Once the breaker was reset, we experienced problems booting the mailserv system (dual HPUX servers). Each server would hang early in the boot process in a way we have not seen before. As a matter of fact, both systems had rebooted cleanly just 2 weeks ago.

We spent a good deal of time trying to determine the stage of the boot process that was being affected. Eventually, we narrowed things down to 3 possible startup shell scripts and we took steps to remove these scripts from the automatic boot process. Once that was done, both systems booted up and we ran the 3 scripts manually with success.

We're still not sure what caused this boot-up problem. We plan to take another look, probably late at night Friday or Saturday. Any related down-time will be pre-announced.

P.S.: WebMail itself was never down. It only appeared to be down for WebMail users who had maildrops on mailserv.

We regret inconveniences caused by this outage.

Nov 13 2002 - Over the past 2 days, we have been having enormous difficulties in dealing with a Denial of Service attack that affected our ability to route traffic off of campus. After working with our two primary ISPs, we have finally been able to resolve the routing issue, and traffic is now leaving campus normally. We are continuing to monitor the situation.

If you are still experiencing difficulties, please contact our NOC at 413-8080 option 2 or send mail to NETWORK@UIC.EDU.

William Marcyniuk
Network Engineer
Academic Computing & Communications Center
University of Illinois at Chicago
Phone: (312) 996-1550
Fax: (312) 996-6834
email: willm@uic.edu

Nov 12 2002 - This has been resolved, very complicated situation initially caused by an outside DOS attack which involved us rerouting how we connect to the outside. ACCC Networks

Nov 07 2002 - Over the past two days the University has experienced degraded network connectivity to the outside world, including UIUC. Initially, this was the result of an unsanctioned and unannounced network experiment between researchers at UIC and a Greek research network which flooded our outside connection. Eventually, we managed to stop the traffic flow from bringing down our outside connections.

In the meantime, our border router's connection to the rest of the campus had become affected in such a way that it was performing dramatically worse than before. After unsuccessfully attempting to fix the issue using a variety of means, including hardware replacement, the decision was made to completely redo the way this router connects to the University backbone. Only then was the problem completely resolved.

However, given the fact that this is a new connection, we will be continuing to monitor performance to insure that service will not be degraded. If performance continues to be an issue, please contact our NOC at 413-8080 option 2 or email us at networks@uic.edu and let us know.

William Marcyniuk
networks@uic.edu