Server hacked (again)

We’ve been hit again by this: http://www.linuxmint.com/blog/?p=235

The good news this time is that we’ll be faster to get rid of it (we’ve got really up to date backups), the bad news is that we’re still obviously vulnerable despite the measures we took the last time. I’ll ask Michael (our sysadmin) to look into this and to find out how this could have happened.

I’ll keep you posted.. I just found out about it a few minutes ago.

Update #1: A backdoor virus was found so it’s possible we got re-infected from the inside. I’m currently re-applying updates to clean the website first.

Update #2: The Wiki, forums, blog, software portal and main website are now clean.

Update #3: I’ll be upgrading the forums to the latest version of phpBb today so they might be offline or disabled for a while.

Update #4: The forums were upgraded to the latest version of phpBB. We’re missing the global announcements and there’s a little problem with the theme but overall they’re back online and they should be working fine.

Update #5: The blog was upgraded to the latest version of WordPress.

Update #6: The wiki was upgraded to the latest version of MediaWiki. We also know more about the problem now.. the first attack left a virus called PHP.RSTBackdoor.

Update #7: The planet was upgraded to the latest version of Gregarius.

Update #8: All the cleanup is done. All our tools were upgraded to their latest versions and we made new backups. Michael identified malware uploaded via mintUpload. We’re discussing the possibility to restrict, secure or even discontinue the free part of this service.

37 comments

  1. Just a shot in the dark, as I have no details, but check for a stub in the boot sector. This has been around for a while but this type of dual infection is difficult to get rid of once it is in place.

    Fred

  2. Same as before. It shouldn’t harm your Linux box, but you may as well run an antivirus just to remove all potential traces of it.

  3. The people who create these pieces of malware code deserve a spot in the lowest circle of hell, right between the pedeophiles and the 419ers.

  4. If you haven’t gotten off Servage,get off ASAP. I see many horror stories about them all the time, especially on Webhostingtalk.

  5. Yes this a virus for the Windows OS. People on Linux should be fine but those who visited the site on a Windows OS should run a virus scan.

    And man am I glad I fully switched to mint right before all of this :P.

  6. i think you should concentrate on the files or folders with 777 permission. cauz most of the time the trouble is from them 🙂

    i hope this could help some how.

  7. ?? It’s run on Gentoo and it’s on a dedicated server. Servage doesn’t do our hosting anymore, they only act as our domain registrar.

  8. U guys r cool for actually telling us, I don’t know of any other web sites that do that for their users.

  9. My Wine had a virus last time this happened…It was in Hotkey.exe…I used avast workstation to get ride of it…love my Minty and I will Protect Her.!! 🙂

  10. @clem

    Once again, i appreciate your candor and honesty about something like this. I have confidence that you and the team will solve the problems and continue the great work you do on Mint. Thanks for a great distro and for your honest & hard work. You guys rock!

  11. The forum is missing the last post subject name.
    Other than that it seems fine.
    I actually have 0 new messages.
    It use to say 4 new messages when I had 0.

  12. You need to re-partition and reinstall. I had this happen and it is almost impossible to find where they broke in and what is infected.

    Install only what you need. Close all ports that aren’t used. Switch to unknown ports such as 29 for ftp instead of 21. Shut down anything anon to get into the server. Change all passwords of course. Make sure you limit your database access for web based programs and don’t put other system passwords into your database. They have found an sql injection in your web software.

  13. I ran the antivirus on windows and calmtk on linux… didnt get anything.. should i be worried?

    i have norton antivirus (2007 or 2006 edition) AND i got zone alarm security suit. (the antivirus in that is disabled cuz i got norton)

  14. It just shows what decent people run the Mint site, that they immediately inform us of the problem. I occasionally access it from work so it is good to have this timely knowledge.

  15. I dunno too much about servers. But somehow these frequent attacks represent that, both, Mint distro and Mint community are growing fast as hell, so people in the other side feel envy and jealousy about us 😀 It must be.

  16. Hmm, I hadn’t thought about the possibility of a Wine virus. Do you have any specific advice for Linux Mint users who browse this site? (That’s gotta be everyone, considering the default home page in Firefox is this page.) “Scan for viruses” is fine advice, but any specific steps to take?

  17. i use GNU/Linux Mint for more than a year and never had problems with viruses,malware,etc,etc. the system is stable and very good!i totally forget about windows! 🙂

  18. I for one installed SELINUX on my Mint just in case. I also installed and use rkhunter. I highly recommend bot. Both are in synaptic.

  19. I realize one probably has nothing to do with the other, but it’s difficult to feel confident in Mint’s security when the website is being hacked repeatedly.

  20. Thanks for he virus warning. I just navigated here and saw it on the page…

    @alessandro
    Your probably right. But why don’t they target Ubuntu.
    Well i guess that Mint is Ubuntu derived.

    Kinda new to Linux but Mint finally got me well and truly in.
    Try a hardware firewall, if thats not already installed.

    j

  21. Your honesty in the matter is greatly refreshing. Still, it is troubling that they were able to do this two times in a row. Hopefully up to date servers will solve the problem and it wasn’t an underlying vulnerability in apache, PHP or mySQL.

  22. I don’t understand, if the site is hosted on a gentoo box and not a windows server, then why would it be a problem?

    Also, why not run on Mint?

    I’m new to Mint but I sure do love it so far!

  23. It’s not a quite simple matter. The safety for a system running Mint is in no way compromised by this. Unless you run Wine nothing has happened, Wine as well as Windows can get hit.
    And the server was not hit, no security hole in it was used.
    It is an SQL injection, meaning that commands where sent that tricked the server to execute “dangerous” commands, and the server “believed” they were issued by the right people. Somehow we had not secured ourselves completely against that. Unfortunately it is possible to do just about anything in this situation and a trojan was left behind that was not detected.
    The origin is a bot that “trawls” the net looking for vulnerabilities – any server can get hit by this (Linux, Mac, Windows)
    But of course our credibility is hit by this, marginally I’d say as we share what happened with the community.

  24. Can’t you prevent SQL injection?? It’s a relatively simple matter once you identify query string concatenations and entry points. In fact you could skip that and just use the PHP function for filtering SQL escape characters on all input.

  25. Yes. We’ve audited all our code after the first attack. And we’ve upgraded all the PHP software we use to their latest versions after the second one 🙂

    I don’t think the second attack came from outside though.. I think the first one left a backdoor. Also, we’ve identified vulnerabilities with mintUpload.

  26. Clem, thanks for the update. I for one will continue to use and endorse Mint for my windows customers making the switch. I’m no expert on SQL, but what SQl database are you using? Would switching to something like Oracle 11g prevent this? I’m sure it’s expensive as all get out. But, might be a worth while investment.

  27. It’s not the database, but the code used to “manipulate” it that is the cause here – and Oracle – well they are definitely not in a hurry to patch….

  28. Clem, do you use any sort of application level firewall – mod_security can be very useful for protecting against this sort of thing. Obviously in addition to a regular firewall. Obviously if you are sure your PHP is now secure, then you can do with out ModSec, but it is a very useful extra layer of protection.

Leave a Reply

Your email address will not be published. Required fields are marked *