Stunning (but secret) Mobile Linux Project

Round Pegs Through a Square Hole

2010-01-28T06:26:00.000-08:00

Hardware integration is moving rapidly. You know, spinning around, hopping about, and generally just.. moving... rapidly.

One of our most recent hiccups has been integrating our wonderful application suite with the Window Manager. Now, don't get me wrong, we have tried to avoid an unhealthy relationship with the whole contraption, but we are limited by it wherever it has not been set up according to our liking.

One of the things we like to do is moving application windows outside the screen area, i.e. move them over the edge of the screen. This is kind of like "pushing the boundaries", or "expanding our horizons", but really its really more like ignoring the boundaries altogether...

Now you might think it strange that we should do something like that. You may say: "how are applications useful when they are located outside the screen?" The answer is, that they are brought into view again when needed. And how? Well, using a very sophisticated system, of course. And sophisticated systems R us!

This brings us to the actual topic of this entry. How to manage application windows on the screen.

As mentioned before, the system relies on XML/TCP/IP connections where IPC is required, so we already have a method for having applications communicate with each other. Secondly, we have an application development platform (Qt), which allows widgets to be moved and hidden and whatnot.

Lets begin by listing the goals that we have for window management:
- moving windows
- animating (interpolating) the locations of application windows on the screen (which is really just an application of the above)
- bringing application windows to the foreground
- hiding application windows

Now, most platforms allow application processes to handle these for themselves: they may move their windows all around the screen, bring their windows to view and hide them again. Of course, with limitations. Additionally, there is almost always a mechanism for actually "managing" all the windows on a screen in a more authoritarian manner. This is often called the Window Manager.

What we have done in our implementation, is we've used a traditional Linux Window Manager. But this doesn't have to be, and indeed will not be in some of the ported versions of this software. Instead, the moving around of windows will be performed by
the individual applications, and overseen by a process that has - for the sake of safety - the ability to kill other application processes if they start misbehaving.

All this can actually be done without touching a window manager at all! The applications are all built in a framework, which allocates windows in the first place, and thus there is great control over the goings on of application windows from there. This can, then, be used via IPC from the "Central", which is and has always been, the core of our system.

Now the mind blowing part of all this is that it allows for unparalleled portability: what system can actually boast of a full windowing system that does not require native components to perform that task?

Of course, to be fair, the applications do use native routines implicitly through the Qt libraries, but that is somebody else's headache :-)

To conclude, I wanted to share one particular example of an application window residing and appearing from outside the device's screen: the virtual keyboard.

In this example, the "TextInput" application makes a kind request for the virtual keyboard to appear. This is routed through the "Central" component, which in turn tells the virtual keyboard it should appear.

Currently the implementation is done using the Window Manager (authoritarian way), but could just as well be implemented using Qt's QWidget::move() function.

GUI Architecture Revisited

2010-01-13T00:57:00.000-08:00

The GUI architecture is designed to allow several separate applications, running in their own processes obviously, to function together as a cohesive UX system.

Secondly, the applications are expected to use platform services through a single point of access and API.

To achieve this, a TCP/IP communications method is used. Control messages (related to the GUI), service discovery, and service usage is all done through the same channel. To the application programmer, this is not revealed, but instead a collection of APIs are provided. The code for XML messaging over TCP/IP is contained within the code of each service.

So, an application will tell the GsmService client instance to dial a number, the GsmService's client-side code will send an XML-message (via TCP/IP, through "Central") to the actual GsmService process. The GsmService process then uses the underlying platform's native services (via D-Bus in the case of moblin, through CTelephony on Symbian).

Graphically, it is traditionally difficult for separate applications to interact in a seamless way. In Windows Vista, this is achieved by having everything work through HW acceleration. This allows for windows to be rotated and skewed, with their contents still updating.

In a mobile software stack, especially a portable one, the problem of drawing several elements on top of each other in a seamless way, is a more difficult one to solve. In our architecture, it is done by sending offscreen pixmaps via TCP/IP to the HomeScreen. Those offscreens can then be rotated, skewed, overlayed with something else or whatever. Seems to work very nicely!

Below are a couple of shots with separate apps running side-by-side. In the left one, a dialer application sits on top of the HomeScreen application, which draws the card boundaries. On the right, an application is partly covered by the virtual keyboard application.

First look at the GUI Applications!

2010-01-12T04:41:00.000-08:00

Ok, the day has arrived. Last year I promised some "swashbuckling" UI-imagery, but was prevented from uploading any concept photos then. But your wait is over now!

This is the first screenshot, but I'll post more in coming days - I promise :-)
The below image depicts the "main view", with new tweets shown as little "cards"... The contents of the tweets are not mine/ours...

The image is pretty much self-explanatory, except for the neat background, which calls for a special mention: the background represents the current time of day (color brightness) and weather (rain drops) dynamically. It can show clear weather, snow, or rain.

Little by little, we are getting closer to having a complete system running. All of this is, of course, in preparation for an upcoming mobile congress..

Qt, GLWidgets, and the Hardware!

2009-12-17T01:21:00.000-08:00

Hi guys!
Its been some time now, since my previous post. And a LOT has happened since!

Two of the most important events have been the release of Qt 4.6 and obtaining the Hardware platforms we will be using for further development.

One of the big questions has been the performance of Qt applications on a mobile device. This is something we obviously haven't tested yet, as the operating system package for flashing is only being compiled as I write, but I am hopeful that if our HW vendor can get its driver act together, we'll have a beautiful and smooth UI in the making here.

A month ago I, as the project's architect, went through a slightly stressy phase related to the graphics. You see, there are really two ways of doing the graphics: OpenGL+Qt drawing within a GLWidget, or by using the OpenGL graphics system for the entire Qt drawing. The GLWidget route has been the plan from the beginning, but some team members expressed their grave concern over the smoothness of graphics (they have since changed their minds, and accepted the architecture allows for great graphics), but in any case, I had to go thought the options.

What I did, then, was go through the options...

Using GLWidget has a strange feature, or maybe it should be called a bug: transparency does not work from GLWidget to GLWidget. The workaround is to implement draw and interaction event passing yourself, and draw into the same (top level widget's) QPainter. This allows for fast graphics, but not really the kind of programming experience that you would like.

Getting back to the team on this, we decided to stick with Qt's opengl rendering via "-graphicssystem opengl". This seemed like a great idea, especially since Qt 4.6 became final that same day!

The tension mounts, as we wait for good HW drivers for the hardware. This pretty much makes or brakes the whole system, as far as graphics are concerned.

Dialer UI & First Actual Phone Calls

2009-11-05T04:11:00.001-08:00

Hello again guys!
Winter is slowly setting in, with the first sporadic snowflakes falling as I write this.
Telephony now works! We have the entire software stack ready to use, and with the dialer UI I have now seen for myself.

The architecture allows for very simple usage of system routines, for example:
1) Dialer UI has SLOT "Dial", Which is Called When the User Clicks on "Call"

void Dialer::Dial( QString aNumber )
{
gsmService->dial( aNumber );
}

2) The "dial"-Function Sends a Message via TCP/IP to the Service Process in Question
This is where the service-specific stuff lives - the "service adaptation" layer. For each service there is an actual service process, and a client service class, such as "GsmService", that the client application can use. Simple, no?

3) The Service Process uses the System's Actual Service
In many cases this involves doing platform-specific things, such as using a service via D-Bus, as is the case in GsmService. If we want to port the system onto another platform, such as Symbian OS, we only have to rewrite this part

4) Yes, it can be this simple :-P

Anyhoo, I can now make phone calls directly from the "Dialer UI", using ofonod running on the system and a phone attached via USB. Very encouraging! Hopefully I'll be able to post pictures of this later on...

2009-10-27T06:35:00.000-07:00

October... UI design, UI development, Hardware RFQs, Trying to tie widgets to services spanning TCP/IP, D-Bus and BEYOND!

Ok, ok, lets turn down the dramatics here. But the truth remains that this project currently feels like a point-and-click adventure game.

As most of my architectural drawings have already been finished, I've had time to work on some GUI implementation issues... such as the dialer UI. The dialer UI goes into the GUI framework, which in turn goes into the app system via TCP/IP. This way the application can use services provided by the "Central", without worrying too much about the underlying "core" implementation.

Currently, due to an epic.. err... chain of events, I have to wait some hours before I can use the dialer system from my beautiful dialer UI.

'nuff said, stay tuned for some swashbuckling UI-imagery later on... this year.

Qt tests

2009-09-16T01:48:00.000-07:00

UI design is banging away at their molten iron, and we are busy trying to keep up with them :-) Ideas are being tossed both ways, and the final (G)UI is on its way!

I can't reveal any secrets, even about my own ideas, at this point, but stay tuned.

Mikael Laine's Qt drawing test:

Timo Holopainen's Qt/OpenGL motion blur effects:

Back from the holidays

2009-09-11T03:37:00.000-07:00

Ok, the last part of my summer holidays is now over, and its back to work for me. This week has been full of meetings, most notably our UI-design people!

Currently we don't have the UI-design yet, but we will have a draft ready in about a week! Right now, the project has been split into two teams/projects: the so-called "core" team, and the "gui" team, of which the graphical designers are a part.

The CORE team is (obviously) very much involved in implementing the below architecture, with its low-level services and other doodads...

The GUI team, on the other hand, is brainstorming the design and figuring out how to actually draw the Qt-widgets etc. using HW acceleration and different effects.

All kinds of demos, protos and other neat graphical stuff abounds, I'll try to post something concrete later on - stay tuned :-)

Desktop widgets

2009-07-29T08:17:00.000-07:00

After some holidays, a move to moblin, and some days at work:

The above image says it all... well, maybe not quite :-)

The architecture is coming along little detail by little detail. Here is one more.

Previously we've had the "Central", which is responsible for launching other apps, and the other apps themselves are pretty much normal qt-apps.

So what I've done now, is I've added the possibility for apps to have a "widget presence" on the "Central's" desktop (the weather icon in this case). This allows for at-a-glance information to be sent to the main view by all applications.

Some kind of priorization/permission system needs yet to be conceived, but as it is, its looking quite nice! I guess the user could be in charge of desktop widgets...

2009-06-25T23:17:00.001-07:00

Long time no post. We had a frantic sprint planning session last week, and the result was that we ended up dumping PokyLinux and moved to moblin!

2009-06-15T00:21:00.000-07:00

Voila! There it is. We're five days into our first sprint (the first sprint is much shorter than normally, for several reasons), and we've pretty much met all our requirements already. "Hello World!" has been touted loud and clear on many a qemu-screen, and all is advancing well!

The above figure illustrates the current architecture draft. In many ways the system needs to be reconsidered, but essentially the architecture is final.

One interesting feature of the architecture (not shown on the screen) is an application lifecycle model, which is built into the base class of all apps on the platform. This should make work easier for the developer.

1st Sprint And the Spirits are High!

2009-06-05T03:35:00.000-07:00

Hello World!
Those are precisely the words our new mobile linux platform should be able to say after this first sprint. Its not that we haven't done that before, but its just that a new project with new requirements and a whole new bunch of people needs a starting point.

A little history: we had a project back in 2007 that created a mobile Linux platform with several applications (DVD-H tv app, email, SMS, browser, dialer/GSM, etc.) but somehow the results just evaporated into the air. A lot of good work was done, but it was not really leveraged. We are now set to change that!

After 6 months, we're going to have a mind-blowing GUI with all the services running that a mobile phone can need. All this will be done on top of a highly portable and modular UI/services platform that will catch the attention of the world.

My name is Mikael Laine, and I'm the chief architect of this secrect project. I'll be letting you in on how everything advances, with screenshots and links to various projects we use and/or modify to our needs.