Maemo.org server move

January 6, 2010 at 12:31 | Posted in Uncategorized | 11 Comments

Ok,

Everyone has seen that the servers at maemo.org are not up to the task anymore. Yes we know and people, especially Ferenc, Niels, Jeremiah, Eero and Ed have been working on the new infrastructure for some time now.

Moving maemo.org has taken some time due to the interwoven nature of the services in maemo.org. Especially the core services for developers (main site, garage, git, builder, repositories) are tied to each other on various levels. These ties have been cleaned during the move to make things ready for the future too. Also those ties mean that moving one part at a time is hard.

Some background

The current server infra was taken into use several years ago, when Maemo was becoming something more than a cool idea in a lab at Nokia. Back in those days the visitor numbers were small and the site was much different. Many of the services did not exist and the ones that did weren’t so popular as they are now.

Then the site slowly grew and more people found it. To accommodate this, some changes were made to the hardware, the main maemo.org site got better servers and the database back-end was updated. This was ok for the majority of last year.

Then Nokia introduced the N900.

As plain statistic, in June 2009 only the main site maemo.org (not including talk, wiki or anything else, just the main site) got roughly 2.5 million page hits a month. This changed to 40 million a month by December 2009.

We expected a growth in traffic, but not a 15 fold increase. The site could have taken about half of that and still performed.

The plan to update the servers was in place early, but unfortunately it was not pushed into action early enough. That’s pretty much my fault, I made the decision to postpone the move until after the Maemo Summit. With 20/20 hindsight: bad decision.

As it happened the ISP that was chosen could not deliver everything in the schedule they optimistically promised. Again my mistake for not scrutinising their delivery capability.

The delays piled up and we ended up in the middle of a massive growth in usage, for which the hardware was not designed for. Services slowed to a crawl like everyone can see.

While things started crawling we had the additional problem of keeping the old system somehow going and building the new one at the same time. No one wants to do that. Double work is always useless.

When will this all be better?

Hopefully next week.

There are two things in the way anymore. The first one is ssh (scp) connections for git and dput. To make things cleaner, we need one more machine to handle incoming ssh traffic, and that is still in the works. The other thing is a complex issue involving a firewall, a load balancer, apache and ssl connections. There’s a solution for that, but it still needs a bit more problem solving to make all the pieces fit together.

Some things have moved already. You might have noticed the mails from Ferenc stating that the list servers are on the new infra. Yes, after some last minute changes (done after the dns change was put through) the mail and list servers are now in the new premises. You might have also noticed that the builder suddenly got faster and your packages don’t spend hours in queue any more. The builder is now a machine that is more up to the task. Also the maemo.org test environments are set up in the new hardware and in use there. Those just aren’t seen by the public, as they are used for testing changes to the site (no, you do not change the live site without testing first).

And of the things that have not moved, many are set up in the new environment. The thing is that moving services requires that the databases move too. And if the databases move, then the services depending on them need to move… Which leads to the situation where most things will move in one big dns change.

So this week you will still see slowness and crawling servers, but next week that should be gone.

I’ll inform when the big dns change is going to hit the site. During a period of roughly 3-12 hours after the change there may be some erratic behaviour, but after that things should calm down.

Thanks for your patience up till now. Hang on a bit longer and things will improve.

About these ads

11 Comments »

RSS feed for comments on this post. TrackBack URI

  1. Guys, the best of luck, hope no last minute gliches pops up, that you will have a smooth change.

  2. Thanks for all the hard work! Also hoping for a smooth transition.

  3. Any idea on how many unique visitors are there? Would be cool to know how many people have or are interested in the N900/Maemo.

    • hello
      i get N900
      i upgrade from N97 To N900
      look like no software yet for N900

      really get pist off

      need the Maemo 6. soon……

      • Hi Herman,

        Please enable the maemo.org repository from the application manager. Then click the download icon in there.

        Also as said before the Ovi store is coming. I know that has been said for a while, but from what I understand it is real soon now.

  4. Thanks for putting the slowness of maemo.org in perspective. Looking forward to the new and snappy infrra next week. good luck!

  5. hi guys, a quick question, would you consider having http://maemo.tv on your server. It’s an ad-free video resource for all things maemo and more. what are your thoughts on this?
    regards, ani

    • My thoughts are that this is the completely wrong place for a question like this.

      Try the community mailing list.

  6. All information entered in bugs.maemo.org yesterday January 9, 2010 – under My Bugs both by me and others – is now GONE! Is that due to the server move??

    • Hi Jon,

      Seems like you are not the only one.
      We’ll look at this, it looks like a side effect of the move.

      Tero

  7. It looks like lots of the bugs.maemo.org database is coming back online, thanks to Andre. Hopefully all the data will be recovered. :)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com. | The Pool Theme.
Entries and comments feeds.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: