torstai 28. tammikuuta 2010

Round Pegs Through a Square Hole

Hardware integration is moving rapidly. You know, spinning around, hopping about, and generally just.. moving... rapidly.

One of our most recent hiccups has been integrating our wonderful application suite with the Window Manager. Now, don't get me wrong, we have tried to avoid an unhealthy relationship with the whole contraption, but we are limited by it wherever it has not been set up according to our liking.

One of the things we like to do is moving application windows outside the screen area, i.e. move them over the edge of the screen. This is kind of like "pushing the boundaries", or "expanding our horizons", but really its really more like ignoring the boundaries altogether...

Now you might think it strange that we should do something like that. You may say: "how are applications useful when they are located outside the screen?" The answer is, that they are brought into view again when needed. And how? Well, using a very sophisticated system, of course. And sophisticated systems R us!

This brings us to the actual topic of this entry. How to manage application windows on the screen.

As mentioned before, the system relies on XML/TCP/IP connections where IPC is required, so we already have a method for having applications communicate with each other. Secondly, we have an application development platform (Qt), which allows widgets to be moved and hidden and whatnot.

Lets begin by listing the goals that we have for window management:
- moving windows
- animating (interpolating) the locations of application windows on the screen (which is really just an application of the above)
- bringing application windows to the foreground
- hiding application windows

Now, most platforms allow application processes to handle these for themselves: they may move their windows all around the screen, bring their windows to view and hide them again. Of course, with limitations. Additionally, there is almost always a mechanism for actually "managing" all the windows on a screen in a more authoritarian manner. This is often called the Window Manager.

What we have done in our implementation, is we've used a traditional Linux Window Manager. But this doesn't have to be, and indeed will not be in some of the ported versions of this software. Instead, the moving around of windows will be performed by
the individual applications, and overseen by a process that has - for the sake of safety - the ability to kill other application processes if they start misbehaving.

All this can actually be done without touching a window manager at all! The applications are all built in a framework, which allocates windows in the first place, and thus there is great control over the goings on of application windows from there. This can, then, be used via IPC from the "Central", which is and has always been, the core of our system.

Now the mind blowing part of all this is that it allows for unparalleled portability: what system can actually boast of a full windowing system that does not require native components to perform that task?

Of course, to be fair, the applications do use native routines implicitly through the Qt libraries, but that is somebody else's headache :-)

To conclude, I wanted to share one particular example of an application window residing and appearing from outside the device's screen: the virtual keyboard.



In this example, the "TextInput" application makes a kind request for the virtual keyboard to appear. This is routed through the "Central" component, which in turn tells the virtual keyboard it should appear.

Currently the implementation is done using the Window Manager (authoritarian way), but could just as well be implemented using Qt's QWidget::move() function.

keskiviikko 13. tammikuuta 2010

GUI Architecture Revisited

The GUI architecture is designed to allow several separate applications, running in their own processes obviously, to function together as a cohesive UX system.

Secondly, the applications are expected to use platform services through a single point of access and API.

To achieve this, a TCP/IP communications method is used. Control messages (related to the GUI), service discovery, and service usage is all done through the same channel. To the application programmer, this is not revealed, but instead a collection of APIs are provided. The code for XML messaging over TCP/IP is contained within the code of each service.

So, an application will tell the GsmService client instance to dial a number, the GsmService's client-side code will send an XML-message (via TCP/IP, through "Central") to the actual GsmService process. The GsmService process then uses the underlying platform's native services (via D-Bus in the case of moblin, through CTelephony on Symbian).



Graphically, it is traditionally difficult for separate applications to interact in a seamless way. In Windows Vista, this is achieved by having everything work through HW acceleration. This allows for windows to be rotated and skewed, with their contents still updating.

In a mobile software stack, especially a portable one, the problem of drawing several elements on top of each other in a seamless way, is a more difficult one to solve. In our architecture, it is done by sending offscreen pixmaps via TCP/IP to the HomeScreen. Those offscreens can then be rotated, skewed, overlayed with something else or whatever. Seems to work very nicely!

Below are a couple of shots with separate apps running side-by-side. In the left one, a dialer application sits on top of the HomeScreen application, which draws the card boundaries. On the right, an application is partly covered by the virtual keyboard application.




tiistai 12. tammikuuta 2010

First look at the GUI Applications!

Ok, the day has arrived. Last year I promised some "swashbuckling" UI-imagery, but was prevented from uploading any concept photos then. But your wait is over now!

This is the first screenshot, but I'll post more in coming days - I promise :-)
The below image depicts the "main view", with new tweets shown as little "cards"... The contents of the tweets are not mine/ours...

The image is pretty much self-explanatory, except for the neat background, which calls for a special mention: the background represents the current time of day (color brightness) and weather (rain drops) dynamically. It can show clear weather, snow, or rain.



Little by little, we are getting closer to having a complete system running. All of this is, of course, in preparation for an upcoming mobile congress..

torstai 17. joulukuuta 2009

Qt, GLWidgets, and the Hardware!

Hi guys!
Its been some time now, since my previous post. And a LOT has happened since!

Two of the most important events have been the release of Qt 4.6 and obtaining the Hardware platforms we will be using for further development.

One of the big questions has been the performance of Qt applications on a mobile device. This is something we obviously haven't tested yet, as the operating system package for flashing is only being compiled as I write, but I am hopeful that if our HW vendor can get its driver act together, we'll have a beautiful and smooth UI in the making here.

A month ago I, as the project's architect, went through a slightly stressy phase related to the graphics. You see, there are really two ways of doing the graphics: OpenGL+Qt drawing within a GLWidget, or by using the OpenGL graphics system for the entire Qt drawing. The GLWidget route has been the plan from the beginning, but some team members expressed their grave concern over the smoothness of graphics (they have since changed their minds, and accepted the architecture allows for great graphics), but in any case, I had to go thought the options.

What I did, then, was go through the options...

Using GLWidget has a strange feature, or maybe it should be called a bug: transparency does not work from GLWidget to GLWidget. The workaround is to implement draw and interaction event passing yourself, and draw into the same (top level widget's) QPainter. This allows for fast graphics, but not really the kind of programming experience that you would like.

Getting back to the team on this, we decided to stick with Qt's opengl rendering via "-graphicssystem opengl". This seemed like a great idea, especially since Qt 4.6 became final that same day!

The tension mounts, as we wait for good HW drivers for the hardware. This pretty much makes or brakes the whole system, as far as graphics are concerned.

torstai 5. marraskuuta 2009

Dialer UI & First Actual Phone Calls

Hello again guys!
Winter is slowly setting in, with the first sporadic snowflakes falling as I write this.
Telephony now works! We have the entire software stack ready to use, and with the dialer UI I have now seen for myself.

The architecture allows for very simple usage of system routines, for example:
1) Dialer UI has SLOT "Dial", Which is Called When the User Clicks on "Call"

void Dialer::Dial( QString aNumber )
{
gsmService->dial( aNumber );
}


2) The "dial"-Function Sends a Message via TCP/IP to the Service Process in Question
This is where the service-specific stuff lives - the "service adaptation" layer. For each service there is an actual service process, and a client service class, such as "GsmService", that the client application can use. Simple, no?

3) The Service Process uses the System's Actual Service
In many cases this involves doing platform-specific things, such as using a service via D-Bus, as is the case in GsmService. If we want to port the system onto another platform, such as Symbian OS, we only have to rewrite this part

4) Yes, it can be this simple :-P

Anyhoo, I can now make phone calls directly from the "Dialer UI", using ofonod running on the system and a phone attached via USB. Very encouraging! Hopefully I'll be able to post pictures of this later on...

tiistai 27. lokakuuta 2009

October... UI design, UI development, Hardware RFQs, Trying to tie widgets to services spanning TCP/IP, D-Bus and BEYOND!

Ok, ok, lets turn down the dramatics here. But the truth remains that this project currently feels like a point-and-click adventure game.

As most of my architectural drawings have already been finished, I've had time to work on some GUI implementation issues... such as the dialer UI. The dialer UI goes into the GUI framework, which in turn goes into the app system via TCP/IP. This way the application can use services provided by the "Central", without worrying too much about the underlying "core" implementation.

Currently, due to an epic.. err... chain of events, I have to wait some hours before I can use the dialer system from my beautiful dialer UI.

'nuff said, stay tuned for some swashbuckling UI-imagery later on... this year.

keskiviikko 16. syyskuuta 2009

Qt tests

UI design is banging away at their molten iron, and we are busy trying to keep up with them :-) Ideas are being tossed both ways, and the final (G)UI is on its way!

I can't reveal any secrets, even about my own ideas, at this point, but stay tuned.

Mikael Laine's Qt drawing test:


Timo Holopainen's Qt/OpenGL motion blur effects: