{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/2505","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/2505/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/2505/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/2505/events","html_url":"https://github.com/BVLC/caffe/pull/2505","id":80190958,"number":2505,"title":"MatCaffe3","user":{"login":"ronghanghu","id":6997335,"avatar_url":"https://avatars.githubusercontent.com/u/6997335?v=3","gravatar_id":"","url":"https://api.github.com/users/ronghanghu","html_url":"https://github.com/ronghanghu","followers_url":"https://api.github.com/users/ronghanghu/followers","following_url":"https://api.github.com/users/ronghanghu/following{/other_user}","gists_url":"https://api.github.com/users/ronghanghu/gists{/gist_id}","starred_url":"https://api.github.com/users/ronghanghu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronghanghu/subscriptions","organizations_url":"https://api.github.com/users/ronghanghu/orgs","repos_url":"https://api.github.com/users/ronghanghu/repos","events_url":"https://api.github.com/users/ronghanghu/events{/privacy}","received_events_url":"https://api.github.com/users/ronghanghu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BVLC/caffe/labels/Matlab","name":"Matlab","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-05-24T15:32:09Z","updated_at":"2015-05-29T09:36:44Z","closed_at":"2015-05-29T08:35:35Z","pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/2505","html_url":"https://github.com/BVLC/caffe/pull/2505","diff_url":"https://github.com/BVLC/caffe/pull/2505.diff","patch_url":"https://github.com/BVLC/caffe/pull/2505.patch"},"body":"This PR creates a powerful Matlab interface for Caffe. It has been an issue for a long time in Caffe that Matlab interface is not as powerful as Python interface. #501 (followed by #1913) attempted to resolve this issue by adding more functions. However features in #1913 are still limited (e.g. not allowing creating multiple nets, not allowing training a net), and development of #1913 seems to have been stalled.\r\n\r\nThis Matlab interface of Caffe implements *more features* when compared with #1913, with *comparable or even less code*, by creating wrappers for `caffe::Solver`, `caffe::Net`, `caffe::Layer` and `caffe::Blob` in Matlab interface. In this interface, **almost everything that can be done in PyCaffe can also be done here** (except for a MATLAB_LAYER, which can be a future functionality).\r\n\r\n~~This PR is add-only and non-invasive, which means the old Matlab interface is not removed in case someone still wants to use it. Note: the old Matlab wrapper doesn't work right now due to #1970.~~ This PR removes the old Matlab wrapper as suggested by @shelhamer , but keeps the HDF5 example and modified the image classification example using BVLC CaffeNet.\r\n\r\nSince #1913 is already called MatCaffe2, I'll call this PR MatCaffe3 :)\r\n\r\n## MATLAB\r\n\r\nThe MATLAB interface -- matcaffe -- is the `caffe` package in caffe/matlab in which you can integrate Caffe in your Matlab code.\r\n\r\nIn MatCaffe, you can\r\n\r\n* Creating multiple Nets in Matlab\r\n* Do forward and backward computation\r\n* Access any layer within a network, and any parameter blob in a layer\r\n* Get and set data or diff to any blob within a network, not restricting to input blobs or output blobs\r\n* Save a network's parameters to file, and load parameters from file\r\n* Reshape a blob and reshape a network\r\n* Edit network parameter and do network surgery\r\n* Create multiple Solvers in Matlab for training\r\n* Resume training from solver snapshots\r\n* Access train net and test nets in a solver\r\n* Run for a certain number of iterations and give back control to Matlab\r\n* Intermingle arbitrary Matlab code with gradient steps\r\n\r\nAn ILSVRC image classification demo is in caffe/matlab/demo/classification_demo.m (you need to download BVLC CaffeNet from [Model Zoo](http://caffe.berkeleyvision.org/model_zoo.html) to run it).\r\n\r\n### Build MatCaffe\r\n\r\nBuild MatCaffe with `make all matcaffe`. After that, you may test it using `make mattest`.\r\n\r\nCommon issue: if you run into error messages like `libstdc++.so.6:version 'GLIBCXX_3.4.15' not found` during `make mattest`, then it usually means that your Matlab's runtime libraries do not match your compile-time libraries. You may need to do the following before you start Matlab:\r\n\r\n    export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:/usr/local/cuda/lib64\r\n    export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6\r\n\r\nOr the equivalent based on where things are installed on your system, and do `make mattest` again to see if the issue is fixed. Note: this issue is sometimes more complicated since during its startup Matlab may overwrite your `LD_LIBRARY_PATH` environment variable. You can run `!ldd ./matlab/+caffe/private/caffe_.mexa64` (the mex extension may differ on your system) in Matlab to see its runtime libraries, and preload your compile-time libraries by exporting them to your `LD_PRELOAD` environment variable.\r\n\r\nAfter successful building and testing, add this package to Matlab search PATH by starting `matlab` from caffe root folder and running the following commands in Matlab command window.\r\n\r\n    addpath ./matlab\r\n\r\nYou can save your Matlab search PATH by running `savepath` so that you don't have to run the command above again every time you use MatCaffe.\r\n\r\n### Use MatCaffe\r\n\r\nMatCaffe is very similar to PyCaffe in usage.\r\n\r\nExamples below shows detailed usages and assumes you have downloaded BVLC CaffeNet from [Model Zoo](http://caffe.berkeleyvision.org/model_zoo.html) and started `matlab` from caffe root folder.\r\n\r\n    model = './models/bvlc_reference_caffenet/deploy.prototxt';\r\n    weights = './models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel';\r\n\r\n#### Set mode and device\r\n\r\n**Mode and device should always be set BEFORE you create a net or a solver.**\r\n\r\nUse CPU:\r\n\r\n    caffe.set_mode_cpu();\r\n\r\nUse GPU and specify its gpu_id:\r\n\r\n    caffe.set_mode_gpu();\r\n    caffe.set_device(gpu_id);\r\n\r\n#### Create a network and access its layers and blobs\r\n\r\nCreate a network:\r\n\r\n    net = caffe.Net(model, weights, 'test'); % create net and load weights\r\n\r\nOr\r\n\r\n    net = caffe.Net(model, 'test'); % create net but not load weights\r\n    net.copy_from(weights); % load weights\r\n\r\nwhich creates `net` object as\r\n\r\n      Net with properties:\r\n\r\n               layer_vec: [1x23 caffe.Layer]\r\n                blob_vec: [1x15 caffe.Blob]\r\n                  inputs: {'data'}\r\n                 outputs: {'prob'}\r\n        name2layer_index: [23x1 containers.Map]\r\n         name2blob_index: [15x1 containers.Map]\r\n             layer_names: {23x1 cell}\r\n              blob_names: {15x1 cell}\r\n\r\nThe two `containers.Map` objects are useful to find the index of a layer or a blob by its name.\r\n\r\nYou have access to every blob in this network. To fill blob 'data' with all ones:\r\n\r\n    net.blobs('data').set_data(ones(net.blobs('data').shape));\r\n\r\nTo multiply all values in blob 'data' by 10:\r\n\r\n    net.blobs('data').set_data(net.blobs('data').get_data() * 10);\r\n\r\n**Be aware that since Matlab is 1-indexed and column-major, the usual 4 blob dimensions in Matlab are `[width, height, channels, num]`, and `width` is the fastest dimension. Also be aware that images are in BGR channels.** Also, Caffe uses single-precision float data. If your data is not single, `set_data` will automatically convert it to single.\r\n\r\nYou also have access to every layer, so you can do network surgery. For example, to multiply conv1 parameters by 10:\r\n\r\n    net.params('conv1', 1).set_data(net.params('conv1', 1).get_data() * 10); % set weights\r\n    net.params('conv1', 2).set_data(net.params('conv1', 2).get_data() * 10); % set bias\r\n\r\nAlternatively, you can use\r\n\r\n    net.layers('conv1').params(1).set_data(net.layers('conv1').params(1).get_data() * 10);\r\n    net.layers('conv1').params(2).set_data(net.layers('conv1').params(2).get_data() * 10);\r\n\r\nTo save the network you just modified:\r\n\r\n    net.save('my_net.caffemodel');\r\n\r\nTo get a layer's type (string):\r\n\r\n    layer_type = net.layers('conv1').type;\r\n\r\n#### Forward and backward\r\n\r\nForward pass can be done using `net.forward` or `net.forward_prefilled`. Function `net.forward` takes in a cell array of N-D arrays containing data of input blob(s) and outputs a cell array containing data from output blob(s). Function `net.forward_prefilled` uses existing data in input blob(s) during forward pass, takes no input and produces no output. After creating some data for input blobs like `data = rand(net.blobs('data').shape);` you can run\r\n    \r\n    res = net.forward({data});\r\n    prob = res{1};\r\n\r\nOr\r\n\r\n    net.blobs('data').set_data(data);\r\n    net.forward_prefilled();\r\n    prob = net.blobs('prob').get_data();\r\n\r\nBackward is similar using `net.backward` or `net.backward_prefilled` and replacing `get_data` and `set_data` with `get_diff` and `set_diff`. After creating some gradients for output blobs like `prob_diff = rand(net.blobs('prob').shape);` you can run\r\n\r\n    res = net.backward({prob_diff});\r\n    data_diff = res{1};\r\n\r\nOr\r\n\r\n    net.blobs('prob').set_diff(prob_diff);\r\n    net.backward_prefilled();\r\n    data_diff = net.blobs('data').get_diff();\r\n    \r\n**However, the backward computation above doesn't get correct results, because Caffe decides that the network does not need backward computation. To get correct backward results, you need to set `'force_backward: true'` in your network prototxt.**\r\n\r\nAfter performing forward or backward pass, you can also get the data or diff in internal blobs. For example, to extract pool5 features after forward pass:\r\n\r\n    pool5_feat = net.blobs('pool5').get_data();\r\n\r\n#### Reshape\r\n\r\nAssume you want to run 1 image at a time instead of 10:\r\n\r\n    net.blobs('data').reshape([227 227 3 1]); % reshape blob 'data'\r\n    net.reshape();\r\n\r\nThen the whole network is reshaped, and now `net.blobs('prob').shape` should be `[1000 1]`;\r\n\r\n#### Training\r\n\r\nAssume you have created training and validation lmdbs following our [ImageNET Tutorial](http://caffe.berkeleyvision.org/gathered/examples/imagenet.html), to create a solver and train on ILSVRC 2012 classification dataset:\r\n\r\n    solver = caffe.Solver('./models/bvlc_reference_caffenet/solver.prototxt');\r\n\r\nwhich creates `solver` object as\r\n\r\n      Solver with properties:\r\n\r\n              net: [1x1 caffe.Net]\r\n        test_nets: [1x1 caffe.Net]\r\n\r\nTo train:\r\n\r\n    solver.solve();\r\n\r\nOr train for only 1000 iterations (so that you can do something to its net before training more iterations)\r\n\r\n    solver.step(1000);\r\n\r\nTo get iteration number:\r\n\r\n    iter = solver.iter();\r\n\r\nTo get its network:\r\n\r\n    train_net = solver.net;\r\n    test_net = solver.test_nets(1);\r\n\r\nTo resume from a snapshot \"your_snapshot.solverstate\":\r\n\r\n    solver.restore('your_snapshot.solverstate');\r\n\r\n#### Input and output\r\n\r\n`caffe.io` class provides basic input functions `load_image` and `read_mean`. For example, to read ILSVRC 2012 mean file (assume you have downloaded imagenet example auxiliary files by running `./data/ilsvrc12/get_ilsvrc_aux.sh`):\r\n\r\n    mean_data = caffe.io.read_mean('./data/ilsvrc12/imagenet_mean.binaryproto');\r\n\r\nTo read Caffe's example image and resize to `[width, height]` and suppose we want `width = 256; height = 256;`\r\n\r\n    im_data = caffe.io.load_image('./examples/images/cat.jpg');\r\n    im_data = imresize(im_data, [width, height]); % resize using Matlab's imresize\r\n\r\n**Keep in mind that `width` is the fastest dimension and channels are BGR, which is different from the usual way that Matlab stores an image.** If you don't want to use `caffe.io.load_image` and prefer to load an image by yourself, you can do\r\n\r\n    im_data = imread('./examples/images/cat.jpg'); % read image\r\n    im_data = im_data(:, :, [3, 2, 1]); % convert from RGB to BGR\r\n    im_data = permute(im_data, [2, 1, 3]); % permute width and height\r\n    im_data = single(im_data); % convert to single precision\r\n\r\nAlso, you may take a look at caffe/matlab/demo/classification_demo.m to see how to prepare input by taking crops from an image.\r\n\r\nWe show in caffe/matlab/hdf5creation how to read and write HDF5 data with Matlab. We do not provide extra functions for data output as Matlab itself is already quite powerful in output.\r\n\r\n#### Clear nets and solvers\r\n\r\nCall `caffe.reset_all()` to clear all solvers and stand-alone nets you have created."},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/106758762","html_url":"https://github.com/BVLC/caffe/pull/2505#issuecomment-106758762","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/2505","id":106758762,"user":{"login":"ronghanghu","id":6997335,"avatar_url":"https://avatars.githubusercontent.com/u/6997335?v=3","gravatar_id":"","url":"https://api.github.com/users/ronghanghu","html_url":"https://github.com/ronghanghu","followers_url":"https://api.github.com/users/ronghanghu/followers","following_url":"https://api.github.com/users/ronghanghu/following{/other_user}","gists_url":"https://api.github.com/users/ronghanghu/gists{/gist_id}","starred_url":"https://api.github.com/users/ronghanghu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronghanghu/subscriptions","organizations_url":"https://api.github.com/users/ronghanghu/orgs","repos_url":"https://api.github.com/users/ronghanghu/repos","events_url":"https://api.github.com/users/ronghanghu/events{/privacy}","received_events_url":"https://api.github.com/users/ronghanghu/received_events","type":"User","site_admin":false},"created_at":"2015-05-29T09:36:44Z","updated_at":"2015-05-29T09:36:44Z","body":"Hi,\r\n\r\nYour code is not correct. You still needs to do mean subtraction. This is the same as in old matcaffe interface. In old interface you can load mean either by `caffe('read_mean', path_to_your_mean_protobin)`, or use the ilsvrc_2012_mean.mat, and this is the same in the new interface, although `caffe('read_mean', path_to_your_mean_protobin)` has been replaced by `caffe.io.read_mean(path_to_your_mean_protobin)`. The ilsvrc_2012_mean.mat is provided (kept from old interface) in case you didn't (or don't want to download imagenet example auxiliary files). But for other mean files created by caffe's `compute_image_mean` tool, you should use `caffe('read_mean', path_to_your_mean_protobin)`.\r\n\r\nI did not provide specific mean subtraction function, as it depends on which method you are using. e.g. CaffeNet first resize to 256x256 and takes 10 crops, and however some method resize min(h, w)=256 and take 10 crops from 256x256 central region, and some first resize min(h, w)=256 and takes 10 crops from 4 conor + center (and flips). Instead of providing a series of such image preparing functions, I decided to leave that to uses and illustrate it in caffe/matlab/classification_demo.m for CaffeNet (AlexNet).\r\n\r\nIf you just want to resize to 256 x 256, and do **one center crop** instead of 10 crops, the correct code is\r\n\r\n    model = './models/bvlc_reference_caffenet/deploy.prototxt';\r\n    weights = './models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel';   \r\n    net = caffe.Net(model, weights, 'test'); % create net and load weights\r\n    % reshape input size to take 1 crop instead of 10\r\n    net.blobs('data').reshape([227 227 3 1]);\r\n    net.reshape();\r\n    \r\n    mean_data = caffe.io.read_mean('./data/ilsvrc12/imagenet_mean.binaryproto');\r\n    im_data = caffe.io.load_image('./examples/images/cat.jpg');\r\n    im_data = imresize(im_data, [256 256]); % resize to 256 x 256 \r\n    im_data = im_data - mean_data; % subtract mean\r\n    im_center_crop = im_data(15:241, 15:241, :); % take 227 x 227 center crop\r\n    \r\n    res = net.forward({im_center_crop});\r\n    prob = res{1};\r\n    [~, predict] = max(prob);\r\n\r\nIf you want to do 10 crops, you can follow `function images = prepare_image(im)` in caffe/matlab/classification_demo.m\r\n\r\nSince approach to prepare input can be arbitrary, if you use other methods with its own way of preparing input, e.g. resize to have min(h, w) = 256 and take crops from center + 4 conor, you should implement it yourself."}}