Linux Mint servers

Forums and community website:

The forums were moved to the USA to a new Cloud Server hosted by Rackspace, running Debian Lenny in 64-bit with 4GB RAM.

The community website remains in Toronto, Canada, where it’s now running on its own.

This configuration is more costly than before but this is not a problem considering the level of financial support we’re receiving from the community and the income generated through traffic in the distribution and the websites. During our tests, we found the forums would run smoothly with 8GB RAM and occasionally hang with 2GB RAM. It costs twice as much each time the amount of RAM is doubled on the server. Since we migrated the forums to their 4GB configuration 4 days ago, we only observed one period of downtime which lasted approximately 15 minutes. The rest of the time, the forums were faster and more responsive than before.

As I’m writing this post, there are 127 people connected to the forums, the server is showing a load average of 0.48, 0.42, 0.37 and using 2.6GB of its 4GB RAM.

The database in Toronto hasn’t gone down since the migration and the community website has had no downtime.

Development/test/archive servers:

We recently lost of test server in London which was also acting as our main public archive rsync server.

The public archive was moved to a second new Cloud Server hosted by Rackspace, running Debian Lenny in 64-bit with 2GB RAM.

The test server wasn’t replaced yet. We’re currently talking to Vault Networks (who already provides us with our Blog server in Miami) and Netrino (who is mirroring Linux Mint from the UK) to acquire a new server through sponsorship. If this fails, we’ll just purchase a new one.

We also started to experiment with remote development and remote release-related activities. The build and smoke-test of the LMDE 201101 32-bit ISO for instance was performed remotely in Miami. This allows us to test and modify ISO images remotely without suffering the delays associated with transferring the images back and forth between the server and our personal locations. I mentioned this briefly, but when the local area only provides a 128kbps uplink and uploading 1GB of data can take more than 2 days, this is a huge advantage for us. Most of our maintainers thankfully benefit from a decent broadband connection but this also represents a plus when we find minor bugs holding an RC release. We can then quickly and remotely modify the ISO, fix the minor bugs, and perform regression testing on it without going through another cycle with the maintainer. At the moment we’re only able to do this in 32-bit. The 64-bit server we’ll use as our test server will also be used for this kind of activities.

Our current status:

Our website is hosted in Stasbourg, France, by Nutime for a ridiculously low price. Michael Kaufmann (a.k.a “d00p”), who is also the owner of the Spotchat IRC network, supported us very early and worked in close collaboration with the team. Kuja and Michael regularly look after our hosting and perform administration tasks and monitoring for us and their involvement with Linux Mint goes far beyond hosting. Linux Mint couldn’t have taken off without their help at the time and though we moved most of our infrastructure to other dedicated locations around the World, we’re still in close relationship with them. Our IRC channels are operated for free on the Spotchat network, and we’re paying for a second dedicated server in Strasbourg which is hosting our repositories.

Vault Networks is our biggest sponsor and they’re providing us with a free dedicated server located in Miami. The server is used to host this blog, but also to seed torrents at high-speed during releases and recently to perform remote development and testing in 32-bit.

Our third biggest sponsor, SevenL, also provides us with a free dedicated server located in Toronto. The server helped us a lot when the traffic became more important and the growth of the project meant services had to be split and taken out of Germany. The forums were successfully hosted there for a long time. The server is now dedicated to hosting the community website.

An account with Rackspace was opened to allow us to quickly create cloud servers and allocate resources to them. Though this technology isn’t suited for remote development or testing, it’s extremely versatile and handy for web hosting and quick migrations. We currently pay for 2 cloud servers, one hosting our forums and the other one hosting our public archive. Due to the high cost of this service, it’s hard to say whether these servers will continue to host the public archive and forums in the future or whether we’ll be migrating these to proper dedicated servers in the future, but one thing is for sure, Rackspace is giving us a lot of flexibility here.

I would like to thank Vault Networks, SevenL and Nutime for the dedicated servers they provide to us. We rarely acknowledge their contribution outside the sponsors page on the website and their help is extremely valuable to us. Linux Mint is growing and many people have observed what is now a real infrastructure problem. We’re scaling and our servers are having a hard time dealing with the popularity of our desktop. Of course, this is extremely positive, though it does represent a real challenge. By communicating differently (hosting torrents both in Strasbourg and Miami, and linking to our HTTP mirrors directly from the blog) we managed to successfully release the latest LMDE without experiencing downtime on our website. With the recent allocation of a cloud server, we now enjoy stable forums and community website. It’s a matter of days before we address the situation with the test server and resume our activity for the upcoming KDE edition. And we’ll soon be able to remotely develop and test in 64-bit. It takes time, it’s not our area of expertise, but as you can see we’re tackling the problem slowly but surely and we’re receiving a lot of help and support from these three companies.

Update (2010-01-04): Our server in London is back and should be operational again as our test server within 48 hours.

17 comments

  1. Thanks Clem for providing us the details. It was an interesting read. I hope things all go well for the future, especially with your testing server. I know you’d been having a hard time these days and I appreciate your hard work.

    Regards, Susu

  2. Thanks for the great info, Clem. It’s such a nice touch that you keep us informed on even the small details involved in producing and maintaining Mint and the supporting infrastructure. That’s a big reason why we feel that we are members of the community, rather than just users.

  3. I agree with tdockery whole heartedly. Your interaction with the community is a great positive. I appreciate all you and the rest of the dev do for us.

  4. Thanks Clem for keeping us updated. You have a very enthusiastic audience and we appreciate very much the effort all of you are putting on the project.

    Linux Mint Really Rocks!!!

  5. Well done folks. I was disappointed with the first spin of LMDE as I could not get it to work with any of the suggestions, but the second spin works perfectly. LMDE runs very smartly from the DVD with no issues at all. I like the thought of a rolling distro without the upgrade/new install of new versions as well as the non-reliance on Ubuntu. Good luck with all your infrastructure challenges. I, like most will be watching this space with interest.
    Al.

  6. Recently OpenBSD developer came out with story that the Cryptographic system in OpenBSD had been “backdoored” by FBI for past eight years (since 2003). In this regards, what is the crypto status of the LinuxMint?

  7. If you guys were SMART, you’d build cheap servers, and host the websites yourselves. It would be a whole lot more convenient than paying for servers overseas…

    Then again, nobody things efficiently like I do. :/

  8. Hi Guys,
    Thanks so much for all that info and the distro itself. We are pensioners n Australia ad I have installed Mint on my wife’s computer and its really going great.
    Thanks again.

  9. Thanks Clem,
    Ever since Mint came to us, windo*s was ditched and I now point all my friends to an operating system with none or little hang ups.
    A great job you are all doing. One quick question will you be concentrating on Gnome or will you be looking at Unity?

Leave a Reply

Your email address will not be published. Required fields are marked *