Forums and community website:
The forums were moved to the USA to a new Cloud Server hosted by Rackspace, running Debian Lenny in 64-bit with 4GB RAM.
The community website remains in Toronto, Canada, where it’s now running on its own.
This configuration is more costly than before but this is not a problem considering the level of financial support we’re receiving from the community and the income generated through traffic in the distribution and the websites. During our tests, we found the forums would run smoothly with 8GB RAM and occasionally hang with 2GB RAM. It costs twice as much each time the amount of RAM is doubled on the server. Since we migrated the forums to their 4GB configuration 4 days ago, we only observed one period of downtime which lasted approximately 15 minutes. The rest of the time, the forums were faster and more responsive than before.
As I’m writing this post, there are 127 people connected to the forums, the server is showing a load average of 0.48, 0.42, 0.37 and using 2.6GB of its 4GB RAM.
The database in Toronto hasn’t gone down since the migration and the community website has had no downtime.
Development/test/archive servers:
We recently lost of test server in London which was also acting as our main public archive rsync server.
The public archive was moved to a second new Cloud Server hosted by Rackspace, running Debian Lenny in 64-bit with 2GB RAM.
The test server wasn’t replaced yet. We’re currently talking to Vault Networks (who already provides us with our Blog server in Miami) and Netrino (who is mirroring Linux Mint from the UK) to acquire a new server through sponsorship. If this fails, we’ll just purchase a new one.
We also started to experiment with remote development and remote release-related activities. The build and smoke-test of the LMDE 201101 32-bit ISO for instance was performed remotely in Miami. This allows us to test and modify ISO images remotely without suffering the delays associated with transferring the images back and forth between the server and our personal locations. I mentioned this briefly, but when the local area only provides a 128kbps uplink and uploading 1GB of data can take more than 2 days, this is a huge advantage for us. Most of our maintainers thankfully benefit from a decent broadband connection but this also represents a plus when we find minor bugs holding an RC release. We can then quickly and remotely modify the ISO, fix the minor bugs, and perform regression testing on it without going through another cycle with the maintainer. At the moment we’re only able to do this in 32-bit. The 64-bit server we’ll use as our test server will also be used for this kind of activities.
Our current status:
Our website is hosted in Stasbourg, France, by Nutime for a ridiculously low price. Michael Kaufmann (a.k.a “d00p”), who is also the owner of the Spotchat IRC network, supported us very early and worked in close collaboration with the team. Kuja and Michael regularly look after our hosting and perform administration tasks and monitoring for us and their involvement with Linux Mint goes far beyond hosting. Linux Mint couldn’t have taken off without their help at the time and though we moved most of our infrastructure to other dedicated locations around the World, we’re still in close relationship with them. Our IRC channels are operated for free on the Spotchat network, and we’re paying for a second dedicated server in Strasbourg which is hosting our repositories.
Vault Networks is our biggest sponsor and they’re providing us with a free dedicated server located in Miami. The server is used to host this blog, but also to seed torrents at high-speed during releases and recently to perform remote development and testing in 32-bit.
Our third biggest sponsor, SevenL, also provides us with a free dedicated server located in Toronto. The server helped us a lot when the traffic became more important and the growth of the project meant services had to be split and taken out of Germany. The forums were successfully hosted there for a long time. The server is now dedicated to hosting the community website.
An account with Rackspace was opened to allow us to quickly create cloud servers and allocate resources to them. Though this technology isn’t suited for remote development or testing, it’s extremely versatile and handy for web hosting and quick migrations. We currently pay for 2 cloud servers, one hosting our forums and the other one hosting our public archive. Due to the high cost of this service, it’s hard to say whether these servers will continue to host the public archive and forums in the future or whether we’ll be migrating these to proper dedicated servers in the future, but one thing is for sure, Rackspace is giving us a lot of flexibility here.
I would like to thank Vault Networks, SevenL and Nutime for the dedicated servers they provide to us. We rarely acknowledge their contribution outside the sponsors page on the website and their help is extremely valuable to us. Linux Mint is growing and many people have observed what is now a real infrastructure problem. We’re scaling and our servers are having a hard time dealing with the popularity of our desktop. Of course, this is extremely positive, though it does represent a real challenge. By communicating differently (hosting torrents both in Strasbourg and Miami, and linking to our HTTP mirrors directly from the blog) we managed to successfully release the latest LMDE without experiencing downtime on our website. With the recent allocation of a cloud server, we now enjoy stable forums and community website. It’s a matter of days before we address the situation with the test server and resume our activity for the upcoming KDE edition. And we’ll soon be able to remotely develop and test in 64-bit. It takes time, it’s not our area of expertise, but as you can see we’re tackling the problem slowly but surely and we’re receiving a lot of help and support from these three companies.
Update (2010-01-04): Our server in London is back and should be operational again as our test server within 48 hours.
