Take the 2-minute tour ×

These days there are two main hardware environments for parallel programming, one is multi-threading CPU's and the other is the graphics cards which can do parallel operations on arrays of data.

The question is, given that there are two different hardware environments, how can I write a program which is parallel but independent of these two different hardware environments. I mean that I would like to write a program and regardless of whether I have a graphics card or multi-threaded CPU or both, the system should choose automatically what to execute it on, either or both graphics card and/or multi-thread CPU.

Is there any software libraries/language constructs which allow this?

I know there are ways to target the graphics card directly to run code on, but my question is about how can we as programmers write parallel code without knowing anything about the hardware and the software system should schedule it to either graphics card or CPU.

If you require me to be more specific as to the platform/language, I would like the answer to be about C++ or Scala or Java.


share|improve this question
C/C++ and GCC atomic built-in functions are the most portable languages (no other languages have been ported on so many platforms). GPUs can be used from OpenCL based on C99 (with Java wrappers). Keep in mind that Java and Scala are using C/C++ libraries and compilers. –  Gil Apr 24 '13 at 6:05

6 Answers 6

up vote 5 down vote accepted

Martin Odersky's research group at EPFL just recently received a multi-million-euro European Research Grant to answer exactly this question. (The article contains several links to papers with more details.)

share|improve this answer
So perhaps if programming in the Scala space its good to start with Actors (AKKA) and parallel collections then wait around for them to work out the details for the graphics cards? Assuming they will work it out in 2 years time. –  Phil Jun 17 '11 at 9:39
The link is dead now. –  cic Apr 16 '14 at 20:15

In a few years from now programs will rewrite themselves from scratch at run-time (hey, why not?)...

...as of right now (as far as I am aware) it's only viable to target related groups of parallel systems with given paradigms and a GPU ("embarrassingly parallel") is significantly different than a "conventional" CPU (2-8 "threads") is significantly different than a 20k processor supercomputer.

There are actually parallel run-times/libraries/protocols like Charm++ or MPI (think "Actors") that can scale -- with specially engineered algorithms to certain problems -- from a single CPU to tens of thousands of processors, so the above is a bit of hyperbole. However, there are enormous fundamental differences between a GPU -- or even a Cell micoprocessor -- and a much more general-purpose processor.

Sometimes a square peg just doesn't fit into a round hole.


Happy coding.

share|improve this answer

OpenCL is precisely about running the same code on CPUs and GPUs, on any platform (Cell, Mac, PC...).

From Java you can use JavaCL, which is an object-oriented wrapper around the OpenCL C API that will save you lots of time and effort (handles memory allocation and conversion burdens, and comes with some extras).

From Scala, there's ScalaCL which builds upon JavaCL to completely hide away the OpenCL language : it converts some parts of your Scala program into OpenCL code, at compile-time (it comes with a compiler plugin to do so).

Note that Scala features parallel collections as part of its standard library since 2.9.0, which are useable in a pretty similar way to ScalaCL's OpenCL-backed parallel collections (Scala's parallel collections can be created out of regular collections with .par, while ScalaCL's parallel collections are created with .cl).

share|improve this answer

The (very-)recently announced MS C++ AMP looks like the kind of thing you're after. It seems (from reading the news articles) that initially it's targeted at using GPUs, but the longer term aim seems to be to include multi-core too.

share|improve this answer
Sounds very interesting, but sadly not here now. –  Phil Jun 17 '11 at 9:43
It is here now in Visual Studio 11, and in fact has a fallback solution for multi-core and SSE when a GPU is not present. –  Daniel Moth Feb 14 '12 at 1:48

Sure. See ScalaCL for an example, though it's still alpha code at the moment. Note also that it uses some Java libraries that perform the same thing.

share|improve this answer

I will cover the more theoretical answer.

Different parallel hardware architectures implement different models of computation. Bridging between these is hard.

In the sequential world we've been happily hacking away basically the same single model of computation: the Random Access Machine. This creates a nice common language between hardware implementors and software writers.

No such single optimal model for parallel computation exists. Since the dawn of modern computers a large design space has been explored; current multicore CPUs and GPUs cover but a small fraction of this space.

Bridging these models is hard because parallel programming is essentially about performance. You typically make something work on two different models or systems by adding a layer of abstraction to hide specifics. However, it is rare that an abstraction does not come with a performance cost. This will typically land you with a lower common denominator of both models.

And now answering your actual question. Having a computational model (language, OS, library, ...) that is independent of CPU or GPU will typically not abstract over both while retaining the full power you're used to with your CPU, due to the performance penalties. To keep everything relatively efficient the model will lean towards GPUs by restricting what you can do.

Silver lining:
What does happen is hybrid computations. Some computations are more suited for other kinds of architectures. You also rarely do only one type of computation, so that a 'sufficiently smart compiler/runtime' will be able to distinguish what part of your computation should run on what architecture.

share|improve this answer

Your Answer


