A request to DNS management has been sent to start pointing maemo.org to the new infrastructure tomorrow at 8:00 UTC and it was acknowledged and scheduled.
After the change there is about a 6-24 hour period in which the DNS change propagates all over the world.
To do things in a sensible way and to minimise downtime, the old site will go to read only (off-line for garage) mode tomorrow morning. Then the last changes are synced over to the new site and the new site will then be opened to public.
When you see the new site naturally depends on the DNS propagation speed, but 3-12 hours seems to be normal based on the smaller moves done earlier.
Things you will experience are:
- Read only on the site before and during the move
- Garage will show a ‘we are moving page’
- No wiki editing during the move
- No git and svn work during the move
Things you might experience are:
- Some data loss is possible, but we’ll try to avoid it
- The site will look strange for a while as the changes take place and the static content is pointing wrong for a while
Things you should not have problems with:
- talk.maemo.org Talk will be open to public all the time, it is on separate hardware and not moving
- bugs.maemo.org Bugs moved last week, it’s there waiting for the other services
- Mailing lists moved to the new hardware some weeks back
After the DNS change propagates the services will start appearing normally again, and naturally there will be notices posted.
When everything is done, please be patient with the service, and open bugs on anything that looks odd or doesn’t work. Raising issues on this post, the community mailing list and on talk will also help. But bugs are a sure way to make an issue known (also remember that more bugs is more work for Andre, so please check for duplicates before opening).
And then a huge thanks to the people who have been hard at work throughout the move! Ferenc and Niels have done the bulk of the work. Ed, Jeremiah, Eero and Bergie pitched in. Much appreciated everyone! Naturally not forgetting the people at the new ISP, it was a crash course into the infra needs of maemo.org for them 🙂
So tomorrow we’ll see how the site moves!
You, like I, may have noticed that PR 1.1 is out as of this morning!
It is a big update and if you get a complaint of too little space, you may need to uninstall some applications. If you run into issues, please post a comment here, ask for help on talk.maemo.org thread or the #maemo IRC channel on freenode.net.
You can find the proper change log in the wiki. The changelog includes a full list of fixed bugs that the community reported against the initial product release. A big thank you to everyone who opened a bug!
The flashable images will be available in the usual location, tablets-dev during today. So you can also flash your device, if the SSU update for some reason fails.
Enjoy the new release!
P.S. it seems that talk.maemo.org got overrun by traffic. Need to get that back on-line again.
Sampo’s thesis will revolve around (surprise) writing and porting software for the Maemo platform. All details are still pretty firmly floating in the air, it’s his third workday, and he just got a desk to put his laptop on.
As eCoach was first implemented for Maemo 4 and has been available for Maemo 5 for some time now, it is natural that it will be used as the case for the thesis. And if anyone is wondering, this won’t affect Sampo’s work on eCoach, rather the opposite. We want to have eCoach out there as an open source project in the future too, it’s a popular application and a good example of what the N900 can do.
Good luck to Sampo for the thesis!
Everyone has seen that the servers at maemo.org are not up to the task anymore. Yes we know and people, especially Ferenc, Niels, Jeremiah, Eero and Ed have been working on the new infrastructure for some time now.
Moving maemo.org has taken some time due to the interwoven nature of the services in maemo.org. Especially the core services for developers (main site, garage, git, builder, repositories) are tied to each other on various levels. These ties have been cleaned during the move to make things ready for the future too. Also those ties mean that moving one part at a time is hard.
The current server infra was taken into use several years ago, when Maemo was becoming something more than a cool idea in a lab at Nokia. Back in those days the visitor numbers were small and the site was much different. Many of the services did not exist and the ones that did weren’t so popular as they are now.
Then the site slowly grew and more people found it. To accommodate this, some changes were made to the hardware, the main maemo.org site got better servers and the database back-end was updated. This was ok for the majority of last year.
Then Nokia introduced the N900.
As plain statistic, in June 2009 only the main site maemo.org (not including talk, wiki or anything else, just the main site) got roughly 2.5 million page hits a month. This changed to 40 million a month by December 2009.
We expected a growth in traffic, but not a 15 fold increase. The site could have taken about half of that and still performed.
The plan to update the servers was in place early, but unfortunately it was not pushed into action early enough. That’s pretty much my fault, I made the decision to postpone the move until after the Maemo Summit. With 20/20 hindsight: bad decision.
As it happened the ISP that was chosen could not deliver everything in the schedule they optimistically promised. Again my mistake for not scrutinising their delivery capability.
The delays piled up and we ended up in the middle of a massive growth in usage, for which the hardware was not designed for. Services slowed to a crawl like everyone can see.
While things started crawling we had the additional problem of keeping the old system somehow going and building the new one at the same time. No one wants to do that. Double work is always useless.
When will this all be better?
Hopefully next week.
There are two things in the way anymore. The first one is ssh (scp) connections for git and dput. To make things cleaner, we need one more machine to handle incoming ssh traffic, and that is still in the works. The other thing is a complex issue involving a firewall, a load balancer, apache and ssl connections. There’s a solution for that, but it still needs a bit more problem solving to make all the pieces fit together.
Some things have moved already. You might have noticed the mails from Ferenc stating that the list servers are on the new infra. Yes, after some last minute changes (done after the dns change was put through) the mail and list servers are now in the new premises. You might have also noticed that the builder suddenly got faster and your packages don’t spend hours in queue any more. The builder is now a machine that is more up to the task. Also the maemo.org test environments are set up in the new hardware and in use there. Those just aren’t seen by the public, as they are used for testing changes to the site (no, you do not change the live site without testing first).
And of the things that have not moved, many are set up in the new environment. The thing is that moving services requires that the databases move too. And if the databases move, then the services depending on them need to move… Which leads to the situation where most things will move in one big dns change.
So this week you will still see slowness and crawling servers, but next week that should be gone.
I’ll inform when the big dns change is going to hit the site. During a period of roughly 3-12 hours after the change there may be some erratic behaviour, but after that things should calm down.
Thanks for your patience up till now. Hang on a bit longer and things will improve.
I’m happy to inform that Niels (X-Fade) will be taking on a bit more responsibility in the maemo.org team.
After thinking and discussing with people how to improve the paid work done on maemo.org, I came to the conclusion, suggested by several people, that the maemo.org paid contributors (Niels, Dave, Andre, Jeremiah, Carsten and Reggie) need to have a team lead.
The goal of this change is to make the maemo.org team more focused and work more toward shared goals.
As a team lead Niels will have more responsibility over the target setting of the maemo.org team and also implementation of work. This means more communication and talk between the team, which is a good thing.
This change does not affect the roles in maemo.org, the division of work is clear as it is. Also this should not effect the work of the council in setting the direction.
Congratulations to Niels on the new responsibility!