High traffic on the package repositories

Written by Clem on Monday, December 15th, 2014 @ 12:45 pm | Main Topics

Our main repository packages.linuxmint.com isn’t currently able to serve connections to everybody. This can result in errors, timeouts and delays in apt-get, and in your update manager.

Please switch to a mirror while we fix this situation:

  • From the menu, open “Software Sources” (or type “mintsources” in a terminal)
  • Type your password
  • Click on the combo box beside “Main”
  • Select a server from the top of the list
  • Click on “Apply”
  • Click on “Update the Cache”

Please accept our apologies for the inconvenience. Our traffic doubled since November and we’re now trying to set up a cluster of servers.

59 Responses to “High traffic on the package repositories”

  1. RamBurner says:

    This should let you know that you release a wonderful distribution.

  2. fortran says:

    Thanks for the info. Greetings.

  3. CarlBiologist says:

    No worries just keep on truckin’ the distro is great I can’t say enough good things about it.

  4. Kirk M says:

    You know your distro is really popular when your repo server chokes.

    Good lord, that sounds like a song lyric.

    No problem though. I’ve come to expect this every time a new version of Mint comes out.

  5. Johnny says:

    I’m not surprised that you have seen a huge influx of new users with the release of 17.1. The LM17.x distribution is just fantastic! Congrats on the new users and good luck getting your new servers up and running.

  6. David says:

    I’m on LM 13 “Maya” LTS (Xfce) – slightly different interface. I assume that after I choose a mirror, I can manually “update the cache” by running “sudo apt-get update” from the terminal? (Maya’s a pretty lovely version, too!)

    Edit by Clem: Hi David, yes.

  7. Matze says:

    I love Linux Mint!!! Thanks for the great work!!!

  8. DarrenG says:

    Probably a good idea to do this anyway as the local servers may be faster. Although in my case a mirror in another country is 3x faster than the one in my country (NZ)! 😉

  9. Dude says:

    I don’t see this as a problem. Linux is kicking Microsoft’s butt!

  10. DustyJurney says:

    Clem, thanks for the tip on the repo. I would like to thank you for your badass distro that I keep putting on my my friend’s and family’s PCs. I am happy to provide an outlet, through Linux Mint, that not only functions like a champ but is user friendly enough for everyone to figure out! -Cheers!

  11. Lynx says:

    Oh… Um, I might be partly responsible for this problem. Maybe I’ve told too many people about Mint. >.>

    Edit by Clem: Aahh.. so it was YOU! 🙂

  12. rhY says:

    You guys are kicking ass and taking names! I’m installing Mint Mate in every classroom I’m in, and that’s a LOT of classrooms.

    Thanks for doing what you are doing! KEEP ROCKING!!

  13. zanfi says:

    Maybe it’s due to a massive installs at Microsoft offices ;-(

  14. kelebek333 says:

    Could I change extra repo form mintsource? Is there any mirror for extras.

    Edit by Clem: Hi, no, extra isn’t mirrored. We’re backing the server with a new one, 4 times more powerful though. It might take a few days, once set up it will split the traffic and should solve the issue. Right now extra and packages are also on the same server, so switching packages to mirrors also eases the traffic on extra.

  15. dale f says:

    Clement,
    That’s a great problem to have.
    Well Done!!!

  16. Maycon says:

    This is the first time I see a apology followed by “thanks” and “we love the product” heehehhe. 🙂

    Really nice work on mint 17 and with the soft transition to 17.1. Worked perfectly.

  17. sebastien says:

    @David (#6): I use to be a Min13 XFCE user; believe me, it worth switching to Mint17

  18. Boris says:

    Is it fixes the problem with the server load?

    I just installed cinnamon mint 17.1 and updates are regularly made, and I have not changed the settings and server upgrades, all is still the default.

    Now I’m not sure whether to upgrade the settings to change the server or to leave at the default?

  19. Xornot says:

    I’m not sure why, but on my installation mintsources was not installed by default. I am using Linux Mint 17 Cinnamon. I ran “sudo apt-get install mintsources” and all is now well.

  20. Uncle Geek says:

    Time for a commercial: Did you see the part about “we’re now trying to set up a cluster of servers.” Clem’s such a nice guy. He gets his hardware for free! And the electric company–you just plug these things in and they work and there’s never any bills!

    Back to earth: that’s what the DONATE button is for. I donated a few times, then realized how many times I had installed it and decided to do the (small) monthly sponsorship I could. (That’s why I haven’t been in the list of individual donors since way back.)

    Now–if you have the ability–it’s your turn to help, one time donation or regularly. Clem appreciates the thanks, but the electric company likes cash.

    Edit by Clem: Thanks 🙂 Now, to be fair.. we are sponsored by several hosting companies, so we don’t pay for all the servers we have. Some are free, some are cheaper than they would normally be, and some are the same price for us than for anybody. This particular server is quite cheap for what it does at $175/month. All in all if we look at the number of servers we have and how much it costs us, we can be very thankful to our hosting partners. And then there’s donations and sponsorships, as you mentioned… we’re basically empowered and money isn’t an issue. After getting that new server we’ll consider upgrading the existing one to get to 8 times the performance we had.

  21. P says:

    I belive the reason (that was what I did) was that the upgrade was not available on all the repo. So user purposely went back to the default repo to be able to upgrade. I guess making sure most repo are updated on time next time will solve the issue.

    Edit by Clem: It’s a factor, but with or without that we were getting close to the limit anyway and it was time to scale things up. Ideally, we want to be able to serve everybody even if nobody switches to another mirror. Even if that means slow speeds, at least it should work reliably. In the scope of the upgrade, the important thing was that the target was up to date. We could have crossed-check sources with a list of known up to date mirrors, but as always in IT, when a process is critical, the simpler it is the better. We put load on our server by switching people back to packages.linuxmint.com but I don’t think it was a mistake, if that prevented eventual issues then it was worth it.

  22. Gary Smith says:

    To Who It May Concern,

    I’d like a 32-bit version of Linux Mint with support for multiple CPUs. This may be the reason I’m getting a white screen with almost no visible GUI showing up as a shadow. Have lots of 32-bit applications.

    Yes, I’d like to learn the terminal commands but not that long using Linux.

    Thank you,

    Gary
    ________

  23. alegui says:

    I think that during installation, the user should be asked if he wants to use another repo than the official, in order to have a faster mirror. Of course, he would be totally free to stay with the official one.

    I add to this post another idea which I had some days ago : why not, rather than make separate isos, make a (discrete) opt-in during the installation to make it an OEM installation (with the ability to set-up later the accounts) ? It would save time (to make separate builds), save bandwidth, and you could make this kind of installation in 32 bits, with multimedia codecs and with KDE/XFCE versions.
    Now that’s two small improvements but I think it is this kind of things that makes mint so attracting!

    Thank you,
    Alegui

  24. Matthew Stringer says:

    Thank You Linux Mint 17.1 and windows is old news bye bye windows.

  25. john says:

    Clem i am a total noob can tell me how this problem can affect my pc? Can be troublesome if i do not change mirror/server? How do you fix it?
    How do i change it in Mint 13?

    Edit by Clem: Hi John, don’t worry, it’s temporary, even if you don’t do anything nothing is going to happen. Right now many users are using the same server, so it’s slow and they get timeouts.. that’s all really.

  26. teg says:

    Why not make packages.linuxmint.com a CDN reference or auto select a mirror like http://cdn.debian.net or http://http.debian.net?

    More info at https://wiki.debian.org/DebianGeoMirror

    Edit by Clem: Our point of failure is delivering connections, we’ve 140k concurrent connections these days. Whether they hit a server or a load balancer, that’s a single point of failure and that machine can bottleneck on this. Ideally we’d split our DNS on multiple load balancers which in turn would delegate to a pool of servers. That’s the extra step. For now we think we can solve the situation by simply splitting the traffic on the same domain name.

  27. CaptainDangeax says:

    Bonjour Clem et toute l’équipe de Mint.

    De mon coté, je prends les miroirs Français et donc je n’ai pas rencontré de problème.
    Par contre, en tant que non-anglophone, je souhaiterais que l’ordre des opérations d’installation commence par le choix de la disposition clavier. Ce serait plus simple pour saisir les noms des dossiers d’autres systèmes en multiboot.

  28. Solidyote says:

    Oddly enough, I changed the mirror, but each times I refresh the package list, it freezes trying to access extra.linuxmint.com/dists/rebecca/InRelease..( or something close to that. )

    I guess I really picked the best time to have my whole system fail on me, and have to re-install everything 😛

  29. Lee says:

    During an upgrade I’ve lost Cinnamon and been booted to the terminal. Any ideas how I can change the mirror from the terminal and recover Cinammon?

    Edit by Clem: In a terminal, ‘apt install mint-meta-cinnamon’. To change the mirrors manually, you need to edit the APT sources in /etc/apt/sources.list.d.

  30. Suggestion says:

    Hi Clem,

    Perhaps you could look at setting up an debdelta mirror for advanced/experimental use? This would be great for reducing server loads!

  31. NeoGeo64 says:

    Linux Mint 17.1 is great but has a few annoying bugs and glitches. It’s a new release so here’s hoping that these bugs are fixed over the next few weeks. I beta tested the RC and helped take care of a few bugs but I can’t help but feel as if 17.1 shouldn’t been released until the beginning of 2015 due to the fact it obviously needed more beta testing because it’s still kind of buggy.

  32. Seb T says:

    @Gary (22). 32-bit Mint (as any other distro) should support multiple CPUs just fine so long as the kernel can correctly detect your hardware’s motherboard/chipset. It sounds to me, from your description, that your problem may have more to do with graphics card/drivers. You could try the more recent 3.16 kernel or the “nomodeset” kernel param (google it!).

    @NeoGeo64 (24). It would be helpful if you could be specific about your problems. Just saying “it’s buggy” etc. isn’t likely to get specific issues looked into or addressed.

  33. Surya Shankar Dan says:

    Linux Mint 17.1 (Cinnamon) is probably the best Linux distribution I have used ever. The only gripe is that most of the Linux Mint applets (http://cinnamon-spices.linuxmint.com/applets) are incompatible with cinnamon 2.4 version; worked perfectly with earlier versions.

  34. James says:

    Loving Mint! Keep the good times coming!

  35. NChewie says:

    @NeoGeo64

    Have to agree with Seb T
    You say 17.1 “shouldn’t have been released until beginning of 2015… because it’s still kind of buggy” but don’t give examples.

    Doing a quick google of your feedback for bugs, I found that you had some problem installing with USB sticks that nobody else encountered, and you currently have one minor issue open on GitHub related to an intermittent screen “bleached out” refresh problem after sleep mode for your particular set up if running certain programmes.

    I would consider neither of these to be show stoppers and do not see anyone else confirming these issues.

    There is nothing wrong with highlighting any bugs found – as that is the way to get them fixed, but I think your open issues do not merit a delay for all the other users who do not experience the same issues.

  36. NChewie says:

    @NeoGeo64

    Ok, I see you have posted some other bugs yesterday about Cinnamon unresponsive and “Mouse and Keyboard unresponsive”.

    I withdraw my indignation :-/

  37. karo says:

    Hi!

    Just a non expert question – is the amount of queries the thing that influences it? would it change anything if the mintupdate would check for updates less often? because when i install mint with family and friends it is one of the things i change quite often because there normally is no need for it to check every 1h. once at start up is more then enough for most users / many users don’t even look into how often that is done.

    Just a thought… not looked into it for a while though so maybe that is changed default already in the latest mintupdate versions.

    thanx for all your hard work

    Edit by Clem: Hi Karo, yes it does. First, you’re right.. the frequency is a bit high and doesn’t need to be set that high by default. Not only does it put load on servers but it also briefly locks the APT cache from the user so we should be more pragmatic on this in the future and change the default value. It was talked about for 17.1 but for some reason it wasn’t done yet. Now, with that said, we still want to be able to face traffic. Our infrastructure should never get in the way and it should be able to follow whatever design decision we want to make. Short term that means it needs to scale, we’re at the limit right now and every new significant update involves performance and bandwidth topics (staged background updates last week, rsync/http concurrency on the KDE 4.14 update earlier this year.. etc), we should sit comfortably at 50% resource usage instead and have servers which empower us rather than limit us in our decisions. On the client side (the OS) there are many ideas we can implement to reduce the load, but even though these ideas are valid, they come as extra.. we still need to scale right now.

  38. BigBuggyBastage says:

    Short list of Clem’s problems:

    1. Released of one of the most-successful Linux distributions to date.
    2. Too many users hammering the repository server.
    3. Enough money on-hand for scaling with extra servers and equipment to handle the load.

    I envy you, sir. :3

    FWIW, I’ve been “on and off” Linux since about 2003 (heh, that makes me sound a bit like an addict), experimenting with kernels, distros, and DEs, as time has allowed. None in the past decade can compare to the ease with which 17.1 Cinnamon has been installed *and* used on my latest Z97/i7 system. It was an utter shock how little I’ve had to do to get things working properly, or how I like them to work. Given what a joy it’s been so far, plus the fact you guys will be on the same codebase for the next few years, I think Mint users are in for a wonderful ride.

    Thanks again for all of your effort, and I wish you smooth sailing through the scaling. 😉

    Edit by Clem: Well yes, I’m not complaining.. we’re happy and all, but right now we still have a significant server issue to fix before we can enjoy Christmas (not to mention the newly released Xfce and KDE editions which aren’t going to make things easier). Anyway, I’d like to thank you all for your support and optimism. I came here to give you bad news and you’re all really happy about things.. it’s a bit unsettling 🙂 I’ll post an update when this is solved. Don’t get me wrong, I’m as happy as you are with the high traffic, but only as long as things keep working 🙂 Last night again we were unable to serve all connections, so the situation is indeed quite serious.

  39. kapil says:

    this means more users are coming to linux mint

  40. coffee412 says:

    I have been using linux since Redhat 4.0 days and I have to say that this is the best flavor I have ever worked with. I have turned a lot of people on to it and they have never looked back at Windows. The amount of dedication and hard work certainly shows in this fine work of art.

    I installed LM17mate on my fathers thinkpad so he can get his email and such for work while in Flordia on vacation and he absolutely loves it over windows. Wanted me to install it on all the office equipment up here. Thats a law office of about 16 computers. Iam so seriously considering this and also would like to get it on the company server instead of the windows server package. But have to wait and see. I run a small business and every computer runs linux instead of windows. I also have introduced LM to a lot of customers.

    Thank you so very much for such a great operating system clem. I saw a recent report that 63 percent of servers are now running linux. I feel that now we can take over the Desktop no problem 🙂

    coffee412

  41. Mario Pedro says:

    hi guys… thx for another great release.

    i have a problem tho:

    i can change the upper server (Main) when i click several times on it.
    but not the Base) server…
    so when updating system gets to download lib6c it tries to download from the Base server and it gets stuck.

    what can i do to fix this?

    thx

  42. Miati says:

    I’ve been trying to change the sources but it’s somewhat difficult.
    I keep getting 404 errors for non-existing repositories on some mirrors…

    /rebecca/main/binary-amd64/Packages 404
    /rebecca/upstream/binary-amd64/Packages 404
    /rebecca/import/binary-amd64/Packages 404
    /rebecca/main/binary-i386/Packages 404
    etc…

    then changing mirrors again fixes this

    on other mirrors, I’m seeing updates not available in packages.linuxmint.com??

    This is inconsistent behaviour :\

  43. Dorian says:

    Hi Clem!

    When you have some free time, please read the following sections:
    http://blog.linuxmint.com/?p=2725#comment-113512,
    http://blog.linuxmint.com/?p=2725#comment-113518,
    http://forums.linuxmint.com/viewtopic.php?f=47&t=183974,
    and
    https://bugs.launchpad.net/linuxmint/+bug/140009.

    The problem isn’t the Main (rebecca), but to the Base (Trusty) (from mintsources ver. 1.3.7. Don’t appear the list from Base (Trusty).

    Thank you for your understanding.

    Sorry for my bad English.

    Best regards,
    Dorian B.

  44. Dorian says:

    Errata: https://bugs.launchpad.net/linuxmint/+bug/1400096.

    Sorry.

  45. kull211 says:

    Yoooo espero por la nueva version de LMDE – LMDE 2 Betsy! Dónde está?! 🙁

  46. Mel says:

    I wanted to leave this message in the http://blog.linuxmint.com/?p=2714 link, but the “Comments” section is currently “closed” and I cannot post there anymore 🙁

    I have a SERIOUS issue with Linux Mint (I don’t know if it’s a bug or not but it IS a problem and I’d like some assistance with it if at all possible. And if it IS a bug (or a stability issue. Or whatever it is), I’d like it fixed as soon as possible). (Thank you).

    Short version of the story:

    I run Linux Mint “Cinnamon” 64-bit (this issue that I’m about to describe has happened on Mint 16, Mint 17, & Mint 17.1) on a “LiveDVD” environment, I install Linux Mint (with the “Install Linux Mint” CD-looking icon on the desktop), and when the installation-wizard finishes installing Linux Mint and it tells me that I can either continue to ‘test’ Linux Mint or ‘restart’ the computer, this happens:

    Scenario A: I select “restart now”. This screen happens (almost all of the time) http://picpaste.com/Photo_Dec_19__2_28_50_PM.jpg

    Scenario B: I select “continue testing” and then I restart the computer myself. (By going to the “Applications Menu” (I’m referring to the Linux Mint equivalent of the Windows “Start Menu”), and selecting the “Turn Off/Shut Down” option and then clicking “Shut Down” in the menu that appears). This screen happens (almost all of the time) http://picpaste.com/Photo_Dec_19__2_28_50_PM.jpg

    Scenario C: I select “continue testing” and then I restart the computer myself. (By going to the “Applications Menu” (I’m referring to the Linux Mint equivalent of the Windows “Start Menu”), and selecting the “Turn Off/Shut Down” option and then clicking “Restart” in the menu that appears). This screen happens (almost all of the time) http://picpaste.com/Photo_Dec_19__2_28_50_PM.jpg

    I don’t know if this is a bug or not. But very rarely (like I said before) do I get the screen I *SHOULD* get: a all-black screen with the “Linux Mint” logo in the middle of it, and on the upper-left-hand-corner, words in white-colored text that say something-along-the-lines-of “please remove installation-media and press “enter” to restart the computer”.

    Like I said, I don’t know if it (ie. this. ie. getting this screen http://picpaste.com/Photo_Dec_19__2_28_50_PM.jpg ) is an issue or not Clem, but if it is, is there any way you guys can fix it as soon as possible?

    I have prior experience doing “press-and-hold the ‘Power’ button of your computer to turn off your computer and press the ‘Power’ button again to turn the computer back on” and the results of that are risky to-say-the-least; any computer-user knows that you risk the stability [and security] of the operating-system [and/or of the computer itself] when you do that, and [that] you should properly turn off the computer by using the “shut down” command and properly restart the computer by using the “restart” command.
    Therefore, I really don’t want to do the “press-and-hold the ‘Power’ button of your computer to turn off your computer and press the ‘Power’ button again to turn the computer back on” thingy.
    And, if there is any way [whatsoever] that you guys (ie. the Linux Mint developers) could fix the issue (ie. the one that causes this screen to always (and/or almost always) appear: http://picpaste.com/Photo_Dec_19__2_28_50_PM.jpg ), that would be great 🙂

    Many thanks for taking the time to read this message 🙂

  47. Francois Blais says:

    I’d like to add a suggestion:
    There are so much mirrors worldwide, it’d be a great to be able to exclude countries from the speed test, which can take a loooong time.
    For instance, since I’m in Canada, I’d like to only check servers in Canada and the USA.
    Other people may also want to only check nearby mirrors.

    Thanks, and Happy Holidays everybody!

  48. Zsolt says:

    This does not work for me. When I change the server I get the same error message:

    Sikertelen letöltés: http://ppa.launchpad.net/cokicd/split-lossless/ubuntu/dists/trusty/main/source/Sources 404 Not Found
    Sikertelen letöltés: http://ppa.launchpad.net/cokicd/split-lossless/ubuntu/dists/trusty/main/binary-i386/Packages 404 Not Found
    Néhány indexfájlt nem sikerült letölteni. Figyelmen kívül lettek hagyva, vagy régebbiek lettek felhasználva.

    What can I do? Help please!

  49. Rick says:

    Installed Mint Xfce 17.1 on USB for Acer AspireOne D257 (2Gb ram); System runs much better (and far cooler -) than it ever did with Windows! Kudos to development team!

  50. Jacob says:

    Wow, Mint is the best and people are voting with their downloads. I hope your scaling plans work out. I have moved to mirrors that are geographically close for me which should relieve some pressure.

    Looking into the (near) future of Mint – any plans to dump systemd? Some really worrying stuff has been highlighted over the Christmas week. I love Mint, but my concerns over systemd are growing, rapidly.

  51. NChewie says:

    @36 Mel
    Do you mean that after doing a successful install from liveDVD, all of the options to shut down from that LiveDVD session give an error message for you – and you need to power off to shut down?

    If that is the case, it I do not see it as a major issue – certainly not as a stability/security issue for your machine. If the install is completed, the newly installed system would not be impacted. Your machine should be running from a read-only DVD and from memory – which will be cleared after power off in any event.

    With your installs of Mint 16, 17 and 17.1, has this had ever resulted in any errors on boot up from your Hard Drive?

  52. Kaufhof says:

    I wonder if Clem can give us a progress report on this yet? Yesterday, I very nearly installed some recommended updates from a substitute repository. I can’t remember any of the details as I swiftly decided to restore the default for safety’s sake!

    The repository was very ‘near the top of the list’.

  53. fred says:

    @Kaufhof
    Yes a new server has been set up, online since a week now. Due to DNs propagation time, it took a few days to be ‘visible’ to users, but now it is serving people smoothly.

  54. Kaufhof says:

    @ fred Thanks for the info; unfortunately nothing at all has happened at my end. Does the repo have a different name or something or have there been no updates as yet?

  55. JJ says:

    One idea, ten reams of code? …. but maybe not.

    Would it be at all feasible for Mint to configure updates to geographically nearby mirrors during the install process? Just after you put in a city / timezone would be one place to try this.

    BTW, 17.1 has me in love with Mint all over again. Truly a wonderful release. It now runs on 4 or 5 systems in my house.

  56. Kaufhof says:

    @fred
    Thanks again – everything looks to be OK now.

  57. kylewooten says:

    Love This distro it is solid so far and very easy to navigate.

  58. Govind says:

    Thanks you for this great Linux build. I’ve used RedHat, Centos (yeah, its a server OS strictly), Fedora, Ubuntu itself but the smoothness of Mint is amazing. Keep up the great work.

  59. Clem says:

    Hi,

    It’s been a few days now and things look pretty good. Two servers now serve traffic on packages.linuxmint.com and the traffic on the original server was consequently halved.

    It looks like we’re ready to grow again then 🙂


Trackbacks & Pingbacks