Sign up ×

Given D=(x,y), y=F(x), it seems most machine learning methods only outputs y as a univariate, either a label or a real value. But I am facing a situation that x vector may only have 5~9 dimensions while I need y to be a multinomial distribution vector which can have up to 800 dimensions. This makes the problem really tricky.

I looked into a lot of things in multitask machine learning methods, where I can train all these y_i at the same time. And of course, another stupid way is that I can also train all these dimensions separately without considering the linkage between tasks. But the problem is, after reviewing many papers, seem that most MTL experiments only deal with 10~30 tasks, which means 800 tasks can be crazy and bad to train. Maybe clustering could be a solution, but I am really curious that can anyone give some suggestions about other ways to deal with this problem, not from a MTL perspective.

share|improve this question
As far as I know neuronal nets, bayesian estimators and alike, they all can estimate a vector. The matrix multiplications that are involved will get a bit big, but what do you mean? – flaschenpost May 31 '13 at 22:01
You mean in opt function f= <w,x>+b, I replace vector w into a matrix? That would be very hard to train. – Yitong Zhou May 31 '13 at 22:04 has two values, but the output layer can have many neurons. – flaschenpost May 31 '13 at 22:06

1 Answer 1

When the input is so "small" and the output so big, I would expect there to be a different representation of those output values. You could analyze if they are a linear or nonlinear combination of some sort, so to estimate the "function parameters" instead of the values itself. Example: We once have estimated a time series which could be "reduced" to a weighted sum of normal distributions, so we just had to estimate the weights and parameters.

In the end you will reach only a 6-to-12-dimensional subspace in some sense (not linear, probably) when you have only 6 input parameters. They can of course be a bit complicated, but to avoid the chaos in a 800-dim space I would really look into parametrizing the result.

And as I commented the machine learning that I know produce vectors.

share|improve this answer
I think the bayes estimator is just an abstract description of learning model in learning theory. To be specific, my question lies in , for example, in hierachical bayesian models, many people find that introduce a Dirichilet prior and then use MCMC or E-M to estimate the parameters can be very helpful. But none of them tried high dimensions. Experiments mainly lie in about 20~30 dimensions. So the key issue is how can I describe my problem or transfer my problem into a lower dimension. – Yitong Zhou May 31 '13 at 22:59
I really have to admit that I have not yet tried to vizualize or analyze a 800-dimensional space to find subspaces, at least in our time series we could see it. Maybe you can also just plot it as a curve, left the input, right the output vector, and play a bit. Order input by some column and step through a "video". Look for linear dependent parts. I guess the data is not public, otherwise I would be curious... – flaschenpost May 31 '13 at 23:13

Your Answer


