Last week in QtGStreamer – a recap (and an overview of GStreamer)

This is inspired by the ‘Last week in Krita’ and ‘Last week in KOffice’ series. Boud, the author of these series, mentioned that this is a nice way to “keep everyone who is interested, developers, contributors, users — everyone — up to date on what is happening in the […] community.” I do not promise I will be able to keep up with weekly updates AND still work on development :), but I will do my best to summarize what is happening on a frequent basis.

I guess starting the series with a recap is good. I have worked in this project for a little more than 5 weeks now, so this is going to be the “Week 1-6” edition. Two weeks ago I posted an introduction to the project, and the reaction so far has been mostly positive. I know of at least one KDE project that is trying QtGStreamer already, and this is good because it lets us tweak the API to real-world applications easily, as no official release has been done and there is still flexibility to do so. And because it is important for us to get more people on board we are usually hanging out in the #gstreamer IRC channel on Freenode: please join us there if you have a question, want to help us with development or just need a hand trying QtGStreamer in your application.

Since that last update we have added some of the basic blocks from the GStreamer API that were still missing, including support for Events, Buffers and TagLists. Some of these are already in the master repository and some are still in review, but we are generally in good shape and ready for prototyping. The plan is to work on the next two weeks on stabilizing what is already added and refactoring some of the code that deals with the bridge between Glib/GObject and Qt. George and I have some interesting ideas on how to deal with some of the challenges that face every GStreamer-binding project, as there are a lot of special cases and design choices in the framework that require authors of bindings to be creative and decide where to draw the line: we can give users full access to the engine at expense of making the API complex, or we can hand-hold the developer and maybe limit the inherent flexibility of the engine. It is a tenuous line, but by now we *think* we have a pretty good overview of the main use cases, and are in a good position to try some things.

I realized after reading the paragraph above that it might not make more sense to people not familiarized with how GStreamer is designed, so I will write a “5 minute overview” of the technology. If you have more time I recommend reading the first chapter of the online docs instead. But, in short, the GStreamer framework is pipeline based. The pipeline is composed of different elements, which are GStreamer’s building blocks. See the image below for the pipeline that can be assembled for a simple ogg player:

Simple Player
A simple GStreamer pipeline

In this example each block is a GStreamer element. An element in its simplest form is a box that generates, processes or outputs data: it can have one or more “pads”, which are the ports that make the connections so data can flow from one element to the other. Pads can be source pads (produce data) or sink pads (consume data). There are over 1000 elements in a typical GStreamer installation, dealing with everything from getting data from cameras (a v4l2 element with a single source pad) to decoding VP8 streams (a vp8 decoder element) to displaying data on the screen (several specialized elements that only have one sink pad, including for example the xvvideosink one).

The pipeline concept is what makes GStreamer so flexible. It is not that different from Unix pipelines in design: the idea is that you build complex solutions starting from simple and discrete elements. As this is supposed to be a 5-minute introduction I will not get into much more detail, but it is important to mention that in GStreamer communication events between elements can flow upstream and downstream, and there are facilities that let the elements communicate back to the application and vice-versa using messages and queries, as well for assembling pipelines dynamically.

So in QtGStreamer we give you all the power to link these GStreamer elements as you wish. A real world example might help: let us say you want to display the image of your webcam in a Qt application window. In Linux you can do this by linking the v4l2src to a qwidgetsink, which as the name implies is a QtGStreamer element that can display images and inherits from QWidget.. Actual code for this is something like:

//Creates the pipeline
m_pipeline = QGst::Pipeline::create();
//Construct the source Video for Linux element
QGst::ElementPtr src = QGst::ElementFactory::make(“v4l2src”);
//Construct the QWidget sink element
m_sink = QGst::ElementFactory::make(“qwidgetvideosink”);
//Add both to the pipeline
//Link the source to the sink
//Tell the sink to use one widget defined in our UI file
//Start the pipeline

That’s it: video should now be flowing from your v4l2 drivers directly to your screen. The example above can be expanded: it is typical to include an element called “ffmpegcolorspace” in between the v4l2src and the qwidgetvideosink, to handle color space conversion if necessary. You could also add an element that duplicates the data before it reaches the qwidget, followed by a VP8 encoder element, a Matroska muxer and an element that writes data to a file, and your application would already compress and record WebM video. But this is a quick tutorial, so we will not expand on that right now. The idea is to show that you have a lot of flexibility and QtGStreamer already lets you construct any pipeline you wish using the make and link functions.

But constructing pipelines manually can be complex. This is where higher-level objects come handy: next week I plan to continue this series with an update on the progress of the project and at the same time will write about how GStreamer’s pre-made bins (which are complex elements that internally have their own collection of elements) can be used to easily play/decode/capture almost any kind of media on several platform, as they auto-discover the elements that need to be used and connect them automatically for you.

Disclaimer: the above description of GStreamer’s design is extremely simplified of course, so I recommend checking the documentation if you are really interested in finding out more about its internals.

Renaming KBattleShip and KTron

As you all know, KDE is a project that encourages as much cooperation with upstream as possible. So if a distro has a reasonable request we generally try to find a way to make life easier for them, as this will ultimately benefit all KDE users when they migrate between different environments.

So it is time for us to make life easier for our Fedora friends, as they have been maintaining patches against some of our games for a while. Please help us choose a new name for KBattleship and also a new one for KTron. These will be considered for KDE 4.6, so we need you to act fast! The poll closes on Sunday, November 14th.

Embedded Linux Conference Europe and GStreamer Conference

This week I am in Cambridge getting to know my colleagues at Collabora Multimedia and attending both the ELC and the GStreamer Conference, which were co-locating.
ELC was interesting. There were a lot of talks dealing with user interface and multimedia issues, and I was pleasantly surprised to find out several presenters mentioning Qt, usually in conjunction with MeeGo. I remember that four or five years ago Qt was relatively unknown, at least in the free software events I used to attend. Nowadays this is clearly not the case: a presenter from Alcatel had a lecture where he compared the available solutions for embedded Linux products. It was a nice presentation, covering all the different SoC architectures as well as the frameworks and toolkits. When he reached the toolkit section there was a big chart with 7 or 8 different solutions, and Qt was clearly a contender for the top position.

The GStreamer conference was a pleasant surprise for me as well, as I was not expecting that many developers. There were at least 150 people there from different projects and companies, and the quality of the presentations was great. I was very pleased to meet another Brazilian developer, Luciana Fujii from Holoscopio. She presented about Landell, a GStreamer-based recording and streaming solution that was used for the last FISL conference in Brazil. It looks very interesting and could possibly help us solve some of the challenges related to recording and streaming content for the next Desktop Summit. On-the-fly camera switching and titling are possible, as well as recording and streaming using different settings. It is Python-based and seems very simple to operate.

I also met several other developers working with GStreamer either professionally or in their pet projects. One I found particularly interesting to watch in action was Stefan Kost’s Buzztard: a pretty impressive music composition environment. I recommend a visit to the site: it can work with both samples and generated sounds, has a built-in sequencer/tracker and can even work with legacy Buzz Windows plugins. For some songs Stefan told me that the internal GStreamer pipeline created by Buzz uses up to 500 elements! (mixers, effects, sound generators…) And it all runs at real time: in his machine this was using 25% of the CPU, but of these only 5% were really being used to play the track, as the rest was mostly consumed by the GUI.

Finally, I collected some initial feedback about the current QtGStreamer implementation and possible use cases. I think we are in the right track so far, and I hope we can see the results of this work pretty soon in the form of applications as cool as the ones I have seen in this conference.

QtGStreamer – an introduction

For the past couple of weeks I have been working on a very cool project, one started and lead by one of our brilliant KDE hackers-that-came-from-GSoC, George Kiagiadakis (gkiagia). I remember that the project got my attention when it was first announced. But I was very busy at that time, and could only follow it from a distance, until now.

As the title of this post reveals, I am talking about QtGStreamer, the Qt-style bindings for the GStreamer multimedia framework.

The project has been progressing during the past year, mostly because it is used in George’s kcall. The structure of the bindings is very flexible, as it follows the powerful pipeline model that is the strength of GStreamer’s design without falling into the trap of trying to map the GStreamer API 1:1. This gives us something that is friendly to the Qt/KDE developer, but at the same time allows things that are very difficult (or impossible) to do with the multimedia APIs we have currently, like displaying a webcam and transcoding the video at the same time with a few lines of code, or streaming audio/video from a remote location transparently.

Even if QtGStreamer is already functional for simple applications it is of course not complete. There is still a lot of work to be done. So as part of my job with Collabora Multimedia I have been assigned to help it reach a feature-complete status. This does not mean the whole GStreamer API will be wrapped: we are aiming for something that is directed specifically to application developers, and I hope it will allow us to help developers code better multimedia experiences for both Qt, MeeGo and KDE users.

My work on the project started recently, and there is still much I have to learn. Luckily George has been around to mentor me, and Collabora Multimedia has a LOT of expertise in GStreamer, of course. I spent the first few weeks doing a full review of the existing GStreamer bindings for all languages, and identifying the missing pieces that we needed to provide. Oh, and making stupid mistakes as well 🙂 You can follow the development of course at the current QGStreamer repository, and don’t be shy if you think you can help, or have a specific request. At this time it is important for us to identify the use cases we have not envisioned for QtGStreamer and do our best to provide an easy API to make them possible, so your input is much appreciated.

I plan to blog regularly about the status of this project. For now, I will already answer the inevitable initial questions:

a) No, it is not meant to replace Phonon. Different use cases, different priorities.
b) No, we could not make this work as part of the Phonon-gstreamer backend. Although the Phonon-gstreamer backend could possibly be ported to QtGStreamer in the future, if there is interest in doing it.
c) No, this does not conflict with the QtMobility/MultimediaKit API. Again, the use case is different, and they might interoperate in the future.

My work on this project is being sponsored by Nokia/MeeGo. As you might know, GStreamer is the official media framework for the MeeGo project, much like Qt is the official developer API. So this project aims to provide the link between those two powerful elements in the MeeGo ecosystem. I will keep you posted about our development, and thanks for reading so far!

Collabora Multimedia

For the past 6 years I have been working on several open source projects as a volunteer. But it has been difficult to find the time to continue to do so on a regular basis after my son was born, in August 2009. There are only 24 hours in a day, and these are not enough to care for the baby, continue my work on Tabuleiro (my small commercial software development company), keep in touch with my friends and family and still make meaningful contributions to KDE and other projects.

During my years with Tabuleiro I accumulated a good deal of experience working with multimedia technology for Windows/OSX, and recently I even managed to convince the bulk of my multimedia customers to do LGPL or GPL projects using Qt as well. But a full transition from closed source to the open source development model was always at my mind. At last Akademy I could witness the number of new companies working strictly with open source projects, most of them leveraging Qt as the main development toolkit, and being able to generate revenue for its developers and maintain a healthy relationship with the community.

And then I realized that I really wanted to work full time on open-source development, after 15 years doing commercial software. This was an easy realization: I did not want to cut on the time spent with my son or with my friends, so changing my work was the only other possible solution 🙂

A few weeks ago the opportunity arrived. I am happy to say that I am now working for Collabora Multimedia on a Qt-based project that I hope will be very useful to Meego and KDE. It is really, really cool. But this is the subject of another post…. For now I just want to say that I am loving the opportunity to work with some of my KDE friends at the same company on a daily basis, as well as having the chance to meet several new developers with different backgrounds.

Updates on Linux Educacional

Linux Educacional (LE) is a distro packaged by the SEED team at the Ministry of Education in Brazil. I posted about the previous versions already, you can find more info on the archives here and here.

The LE team has been working on the 2011 version for a while. Previous releases used KDE 3.5, and the next one is going to be the first deployed over a KDE 4.x desktop. Helio Castro and I went to Brasilia a few months ago to field some questions, discuss a couple of design decisions that are important for the transition and to offer the help of the KDE community, if needed. We were very happy to learn that among all the options available the decision was made to stick with KDE as the LE desktop: one of the reasons was the possibility of deep and easy customization offered by the Plasma environment; as well as the need to preserve the existing investment in training for teachers and IT people and the familiarity of teachers and students with the existing KDE-Edu software.

Then, during Akademy, we saw a lot of interest on this project coming from members of the Plasma, Edu and Oxygen teams. The plan was to foster collaboration between the KDE and the LE teams, and if possible to develop most of the customization features they needed upstream. A mailing list and a wiki page were created, and some ideas floated around.

But… working upstream proved to be very difficult for this project. There were language barriers, and also issues related to timing and the decision-making chain. I find it understandable, but personally frustrating. Development is happening mostly at closed doors, as I was told the team is under a very tight schedule. Apparently a portion of the project is now being handled by the computer science department of UFPR, a federal university in the Paraná state in Brazil. Their staff is known for its technical excellence and skill, so I have high hopes for the quality of the final implementation. Some of the people involved have been spotted on the #kde-brasil IRC channel, and the KDE community at large is still available to help of course, if required.

So, even if we could not really collaborate in the true open source fashion for now, it is nice to know that the work we are doing is useful and relevant to the educational community, and that a KDE4-based LE will equip thousands of computer labs next year. And I can see as well that this is a slow learning process, and maybe one day we will have the opportunity of really working together on a global scale on a deployment like this.

Thanks to all members of the KDE community who spent some time during Akademy and in the following weeks thinking about the educational desktop and how to contribute to efforts like the LE customization. I am sure we will have the opportunity to build on several of these ideas when the right time comes. For now, the “we build it and they use it” paradigm remains the most common, and it is not a bad one per se. But I still hope upstream development and collaboration slowly becomes the norm and not the exception.

Blog v3

Well, the last few months have been very busy. And I realized I have not blogged since Akademy. Of course, this is the perfect excuse to change blog hosts again 🙂 Livejournal has been good, but ultimately the amount of spam on comments is just too much, even for a semi-alive blog like mine. So WordPress, here we go, with a new and shining address. As a bonus I collected all the posts (minus comments) from my previous blogspot and livejournal addresses, and archived them here, for posterity.

Let’s get back to Akademy 2010, then. What can I say? Perfect location, perfect organization, and perfect company. It was great to meet old friends again (David! Neja! Jure! lots of people…) as well as make new ones (hi Nuno!.) Too bad I could only stay for 4 days. At the last morning I conducted a BoF about the involvement of KDE with the Linux Educacional project going on in Brazil, an effort that we attempted to jumpstart and that generated a lot of interest inside the KDE Plasma and Edu communities. I will update you guys on how the project is going in my next blog entry.

Akademy 2010

Akademy also marked the very first time I was away from my son for more than a day. It was not easy for me, although I heard that he managed it very well 🙂

Akademy-br and KEduca

Akademy-br is over, and for me it was a great event. I expect the results to roll in over the next couple of months, as several projects have been started. Tomaz and the people working on card games for KDE Games will probably chime in soon, and I know the promo people are preparing the video and photo collections from the event.
I spend some time on the event helping the people that is going to work on the new KEduca. The summary of this project can be read on this message to the KDE Edu mailing list, so I will not repeat it here. Welcome Syst guys, and I hope this collaboration with KDE Edu brings lots of rewards for both our community and your users as well.
Among several interesting conversations and quick hacking sessions, on the last day I had a chance to work very briefly with Luiz Romário, a very brilliand and young member of the KDE community here. He is working on several projects, and I am happy to see his patches finding their way into ReviewBoard and soon into Kiriki.

In CampKDE 2009 (Jamaica) I had already experienced this feeling of a conference/sprint that happens in the same location as the lodging and where food is served. This saves a lot of hours as people do not have to scramble for food or transportation, and the result is that the rooms were busy even in the middle of the night. Not something possible for a big event like Akademy, but something to look for when we scout for locations for the smaller meetings and sprints, imo.

People mentioned the torrential rain that fell over the area. In a sense it was a good, because I never get to the beach during the event,  people were trapped inside the hacking rooms and became even more productive 🙂   And a family of small monkeys fleeing from the rain also stood there in the balcony on day two:

code monkeys
code monkeys2

A quick joke involving rain, sprints and code monkeys is forming in my head, but I can not fully translate it into English, sorry… 🙂

In closing, I would also like to thank Sandro Andrade very much for his work on leading the organization of the event, and also for his huge contributions to the local community. Tomaz also did lots and lots of work there, and at times it appeared that he has mastered the art of being in several places at the same time…. A good portion of the sprint members were introduced to KDE by Sandro and Tomaz and their courses and lectures, and this shows to me that:

a) one person can make a difference
b) even “more differences” can be attained when people work together towards a common goal, and this is what these guys are doing with the LiveBlue user group.