Code diary

QtGStreamer – an introduction

For the past couple of weeks I have been working on a very cool project, one started and lead by one of our brilliant KDE hackers-that-came-from-GSoC, George Kiagiadakis (gkiagia). I remember that the project got my attention when it was first announced. But I was very busy at that time, and could only follow it from a distance, until now.

As the title of this post reveals, I am talking about QtGStreamer, the Qt-style bindings for the GStreamer multimedia framework.

The project has been progressing during the past year, mostly because it is used in George’s kcall. The structure of the bindings is very flexible, as it follows the powerful pipeline model that is the strength of GStreamer’s design without falling into the trap of trying to map the GStreamer API 1:1. This gives us something that is friendly to the Qt/KDE developer, but at the same time allows things that are very difficult (or impossible) to do with the multimedia APIs we have currently, like displaying a webcam and transcoding the video at the same time with a few lines of code, or streaming audio/video from a remote location transparently.

Even if QtGStreamer is already functional for simple applications it is of course not complete. There is still a lot of work to be done. So as part of my job with Collabora Multimedia I have been assigned to help it reach a feature-complete status. This does not mean the whole GStreamer API will be wrapped: we are aiming for something that is directed specifically to application developers, and I hope it will allow us to help developers code better multimedia experiences for both Qt, MeeGo and KDE users.

My work on the project started recently, and there is still much I have to learn. Luckily George has been around to mentor me, and Collabora Multimedia has a LOT of expertise in GStreamer, of course. I spent the first few weeks doing a full review of the existing GStreamer bindings for all languages, and identifying the missing pieces that we needed to provide. Oh, and making stupid mistakes as well 🙂 You can follow the development of course at the current QGStreamer repository, and don’t be shy if you think you can help, or have a specific request. At this time it is important for us to identify the use cases we have not envisioned for QtGStreamer and do our best to provide an easy API to make them possible, so your input is much appreciated.

I plan to blog regularly about the status of this project. For now, I will already answer the inevitable initial questions:

a) No, it is not meant to replace Phonon. Different use cases, different priorities.
b) No, we could not make this work as part of the Phonon-gstreamer backend. Although the Phonon-gstreamer backend could possibly be ported to QtGStreamer in the future, if there is interest in doing it.
c) No, this does not conflict with the QtMobility/MultimediaKit API. Again, the use case is different, and they might interoperate in the future.

My work on this project is being sponsored by Nokia/MeeGo. As you might know, GStreamer is the official media framework for the MeeGo project, much like Qt is the official developer API. So this project aims to provide the link between those two powerful elements in the MeeGo ecosystem. I will keep you posted about our development, and thanks for reading so far!

Mauricio Piacentini

, , , ,

24 responses to “QtGStreamer – an introduction”

    • Well, I am not sure if Phonon was designed for this. To me Phonon’s greatest strength is that it lets you abstract the backend. Of course, this also means you typically do not explore all the capabilities of each backend, and has to work with a subset of functions. For simple audio and video playback it has served us fine. As I mentioned in the post, the goal of QtGStreamer is not to replace Phonon.

      • Well, the backend-abstraction means that you *can* use different backends what a requirement for portability and platform-integration. So, as developer I don’t need to care if my application is using GStreamer, VLC, Direct Sound, whatever to do the job and replacing the backends can be done on the fly without changes in the applications. But as you sayed that has the disadvantage that I need to define a lowest common denominator for all supported backends or I need to work around limitations at certain backends somehow if possible at all.

    • Well, I believe it would be possible once the bindings are complete, but I must say I am not familiar with the existing Phonon gst backend code so it is premature to tell if we should pursue this. The main advantage of QtGStreamer over the C API is of course that it is familiar to Qt/C++ developers, so it might help us find more maintainers and hackers to work on the code in the long run.

  5. This sounds wonderful, must have taken a while to write! I really hope this really becomes the back-end of Phonon-GStreamer. As you already said, that will give KDE applications things like webcam support.

  6. As others have asked, is this really a good idea with Phonon around? Could this lead to users using one back-end for some apps and another back-end for others?

    As I understand it, Phonon gives special properties and capabilities to apps when using different back-ends, such as pulseaudio or vlc. Could the same be done for gstreamer, then?

    • This work has nothing to do with Phonon directly, or with the Phonon-gstreamer backend. People can (and will) continue to use Phonon as they have done until today.
      The scope of the project is different: we aim to give Qt developers an easy Qt-style C++ API that can be used to leverage the capabilities of GStreamer, which already include not only audio and video playback, but also media capture, transcoding and streaming, for example.
      But getting back to your comment, I don’t think application authors using Phonon can rely on capabilities that will be present only on a specific backend. Doing so kind of defeats the whole purpose of having Phonon, doesn’t it? The idea of Phonon is to provide a very high level API that works also on Windows and OSX. The minute developers are tied to one particular backend (VLC or gst for example) then there is no real advantage in having the Phonon layer, which is supposed to shield you from issues like ABI compatibility and so.

    • I don’t know, really 🙂 Probably not, as QtMM already has its own code for backends, including one for gstreamer, written directly for the C API.

  8. Great idea, I am just working on usability tester application and this is a thing which would be very useful for me. I need to record user’s screen, external camera and microphone by using C++ and qt libraries. I am looking forward to your progress with QtGstreamer.

  9. Qt is a delight to write programs with. GStreamer is, architecturally, probably the most flexible and all around best multimedia framework we have. This is wonderful news. Hmm…Qt Declarative + QTGStreamer = awesomely slick media center?

    • Well, I think I could, it is just an option on the feed. But to be honest I would prefer the opposite: if everyone just syndicated an abstract of their posts then the main planetkde.org page would load much faster, and we would then only load the posts we are actually interested in I have to check however with the other places where these feeds are linked to see if this does not violate their policy, or if I can syndicate full posts only to planetkde somehow.

  13. […] QtGStreamer – an introduction For the past couple of weeks I have been working on a very cool project, one started and lead by one of our brilliant KDE hackers-that-came-from-GSoC, George Kiagiadakis (gkiagia). I remember that the project got my attention when it was first announced. But I was very busy at that time, and could only follow it from a distance, until now. […]

  14. Good stuff! Have you given any thought as to how this could eventually be incorporated into QML as an extension? I’m warming up to the idea of using a declarative ui for my next project. It would be nice to extend QML with some solid multi-media capabilities. I think the Trolls are focused on mobile development for now, but this project looks like a nice fit to get video working with declarative on the desktop.

  15. Looks like gstreamer has some functional winbuilds now. Can we expect QtGstreamer to work in a Qt + GStreamer for windows environment?

    • Yes, there is not technical reason preventing this. At this time we are focused on Linux/Unix, but when the work is finished there is interest in making sure it runs transparently under Windows and OSX as well.