Maemo.org server moveJanuary 6, 2010 at 12:31 | Posted in Uncategorized | 11 Comments
Everyone has seen that the servers at maemo.org are not up to the task anymore. Yes we know and people, especially Ferenc, Niels, Jeremiah, Eero and Ed have been working on the new infrastructure for some time now.
Moving maemo.org has taken some time due to the interwoven nature of the services in maemo.org. Especially the core services for developers (main site, garage, git, builder, repositories) are tied to each other on various levels. These ties have been cleaned during the move to make things ready for the future too. Also those ties mean that moving one part at a time is hard.
The current server infra was taken into use several years ago, when Maemo was becoming something more than a cool idea in a lab at Nokia. Back in those days the visitor numbers were small and the site was much different. Many of the services did not exist and the ones that did weren’t so popular as they are now.
Then the site slowly grew and more people found it. To accommodate this, some changes were made to the hardware, the main maemo.org site got better servers and the database back-end was updated. This was ok for the majority of last year.
Then Nokia introduced the N900.
As plain statistic, in June 2009 only the main site maemo.org (not including talk, wiki or anything else, just the main site) got roughly 2.5 million page hits a month. This changed to 40 million a month by December 2009.
We expected a growth in traffic, but not a 15 fold increase. The site could have taken about half of that and still performed.
The plan to update the servers was in place early, but unfortunately it was not pushed into action early enough. That’s pretty much my fault, I made the decision to postpone the move until after the Maemo Summit. With 20/20 hindsight: bad decision.
As it happened the ISP that was chosen could not deliver everything in the schedule they optimistically promised. Again my mistake for not scrutinising their delivery capability.
The delays piled up and we ended up in the middle of a massive growth in usage, for which the hardware was not designed for. Services slowed to a crawl like everyone can see.
While things started crawling we had the additional problem of keeping the old system somehow going and building the new one at the same time. No one wants to do that. Double work is always useless.
When will this all be better?
Hopefully next week.
There are two things in the way anymore. The first one is ssh (scp) connections for git and dput. To make things cleaner, we need one more machine to handle incoming ssh traffic, and that is still in the works. The other thing is a complex issue involving a firewall, a load balancer, apache and ssl connections. There’s a solution for that, but it still needs a bit more problem solving to make all the pieces fit together.
Some things have moved already. You might have noticed the mails from Ferenc stating that the list servers are on the new infra. Yes, after some last minute changes (done after the dns change was put through) the mail and list servers are now in the new premises. You might have also noticed that the builder suddenly got faster and your packages don’t spend hours in queue any more. The builder is now a machine that is more up to the task. Also the maemo.org test environments are set up in the new hardware and in use there. Those just aren’t seen by the public, as they are used for testing changes to the site (no, you do not change the live site without testing first).
And of the things that have not moved, many are set up in the new environment. The thing is that moving services requires that the databases move too. And if the databases move, then the services depending on them need to move… Which leads to the situation where most things will move in one big dns change.
So this week you will still see slowness and crawling servers, but next week that should be gone.
I’ll inform when the big dns change is going to hit the site. During a period of roughly 3-12 hours after the change there may be some erratic behaviour, but after that things should calm down.
Thanks for your patience up till now. Hang on a bit longer and things will improve.