Look for hits on the script that was running in the logs; search again for IPs that hit that script. Try to find out where they came in on the site — did they hit the front page first? Did they hit any pages besides the script? Look for common threads and try to trace your way back to the earliest instances of hits on the attack scripts. See what they accessed just before.

Often times, these “first hits” are POSTs against a legit script on the site. For example, a recent ZenCart (shopping cart software) exploit starts out like this:

/admin/record_company.php/ password_forgotten.php?action=insert

This particular exploit is from:

http://www.milw0rm.com/exploits/9004

And the client ought follow the instructions from ZenCart here:

http://www.zen-cart.com/forum/showthread.php?t=130161

Keep abreast of the hacks as they come out — it’s always cat and mouse, but there’s no reason to lag too far behind!

Examine the output for the user apache running anything but the correct apache binary (/usr/sbin/httpd or /usr/local/bin/apachectl -k startssl). Also look for children processes running with extremely high CPU time (if most children are running with 1:20 or so, and one is running at 60:42, note hte pid of the latter).

Closely examine the child process that is unusual with lsof.

lsof -p 19232

Note any network connections:

lsof -p 19232 |grep IPv4

and note the current working directory of the process:

lsof -p 19232 |grep cwd

It is in this directory that we start the search. Review other files in that directory — look specifically for files owned by apache, files with a much newer modification time than other files in the directory, and hidden directories (named ” ” or “…” etc) or unfairly-named files (”å¥∂øƒø∂ç.php” for example). Investigate suspicious files themselves and use discretion as to whether it is legit or not — it helps to know a bit of PHP or coding here. Note the modification times of the files as well. If you’re certain a file is not part of the site and decidedly malicious, move it away:

mkdir /home/rack/ticketnumber/comp
mv c99.php !!$

Often attackers will hide other shells around the web roots if the apache user can access them (777 dirs). Look for apache-owned scripts to aid the search:

cd /home/rack/ticketnumber
find /home/httpd/vhosts/ -user apache -mtime 365 |egrep "\.php|\.cgi|\.pl" >> newfiles.out

You will have to manually review newfiles.out and each file to tell if a file is malicious. Give the code the benefit of the doubt, but look for tell-tale shells, obfuscated code, and vandalism.

Once files are identified, cull out the legitimate-looking files from the list. Move the malicious files out of the webroots:

alias mv='mv'
cat newfiles.out |while read line ; do mv $line comp ; done

Again review newfiles.out and manually review the directories that contained the malicious scripts. Look for hidden directories, suspicious files that didn’t match the filter (c code, binaries, etc). Remove this content as well.

If you have even the inkling of suspicion, there’s no harm in getting a “feel” for your environment. Note anything that feels unusual; look for anything that stands out. Unless it’s root compromised, there are always breadcrumbs.

Document the environment as it is active:

mkdir /home/rack/ticketnumber
cd !!$
script !!$.out
ps auxwww >> psauxwww.out
pstree >> pstree.out
netstat -plan >> netstat-plan.out
lsof -i >> lsof-i.out
lsof >> lsof.out

Here we determine what kind of compromise it is.