{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/2505","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/2505/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/2505/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/2505/events","html_url":"https://github.com/BVLC/caffe/pull/2505","id":80190958,"number":2505,"title":"MatCaffe3","user":{"login":"ronghanghu","id":6997335,"avatar_url":"https://avatars.githubusercontent.com/u/6997335?v=3","gravatar_id":"","url":"https://api.github.com/users/ronghanghu","html_url":"https://github.com/ronghanghu","followers_url":"https://api.github.com/users/ronghanghu/followers","following_url":"https://api.github.com/users/ronghanghu/following{/other_user}","gists_url":"https://api.github.com/users/ronghanghu/gists{/gist_id}","starred_url":"https://api.github.com/users/ronghanghu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronghanghu/subscriptions","organizations_url":"https://api.github.com/users/ronghanghu/orgs","repos_url":"https://api.github.com/users/ronghanghu/repos","events_url":"https://api.github.com/users/ronghanghu/events{/privacy}","received_events_url":"https://api.github.com/users/ronghanghu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BVLC/caffe/labels/Matlab","name":"Matlab","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-05-24T15:32:09Z","updated_at":"2015-05-30T03:08:09Z","closed_at":"2015-05-29T08:35:35Z","pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/2505","html_url":"https://github.com/BVLC/caffe/pull/2505","diff_url":"https://github.com/BVLC/caffe/pull/2505.diff","patch_url":"https://github.com/BVLC/caffe/pull/2505.patch"},"body":"This PR creates a powerful Matlab interface for Caffe. It has been an issue for a long time in Caffe that Matlab interface is not as powerful as Python interface. #501 (followed by #1913) attempted to resolve this issue by adding more functions. However features in #1913 are still limited (e.g. not allowing creating multiple nets, not allowing training a net), and development of #1913 seems to have been stalled.\r\n\r\nThis Matlab interface of Caffe implements *more features* when compared with #1913, with *comparable or even less code*, by creating wrappers for `caffe::Solver`, `caffe::Net`, `caffe::Layer` and `caffe::Blob` in Matlab interface. In this interface, **almost everything that can be done in PyCaffe can also be done here** (except for a MATLAB_LAYER, which can be a future functionality).\r\n\r\n~~This PR is add-only and non-invasive, which means the old Matlab interface is not removed in case someone still wants to use it. Note: the old Matlab wrapper doesn't work right now due to #1970.~~ This PR removes the old Matlab wrapper as suggested by @shelhamer , but keeps the HDF5 example and modified the image classification example using BVLC CaffeNet.\r\n\r\nSince #1913 is already called MatCaffe2, I'll call this PR MatCaffe3 :)\r\n\r\n## MATLAB\r\n\r\nThe MATLAB interface -- matcaffe -- is the `caffe` package in caffe/matlab in which you can integrate Caffe in your Matlab code.\r\n\r\nIn MatCaffe, you can\r\n\r\n* Creating multiple Nets in Matlab\r\n* Do forward and backward computation\r\n* Access any layer within a network, and any parameter blob in a layer\r\n* Get and set data or diff to any blob within a network, not restricting to input blobs or output blobs\r\n* Save a network's parameters to file, and load parameters from file\r\n* Reshape a blob and reshape a network\r\n* Edit network parameter and do network surgery\r\n* Create multiple Solvers in Matlab for training\r\n* Resume training from solver snapshots\r\n* Access train net and test nets in a solver\r\n* Run for a certain number of iterations and give back control to Matlab\r\n* Intermingle arbitrary Matlab code with gradient steps\r\n\r\nAn ILSVRC image classification demo is in caffe/matlab/demo/classification_demo.m (you need to download BVLC CaffeNet from [Model Zoo](http://caffe.berkeleyvision.org/model_zoo.html) to run it).\r\n\r\n### Build MatCaffe\r\n\r\nBuild MatCaffe with `make all matcaffe`. After that, you may test it using `make mattest`.\r\n\r\nCommon issue: if you run into error messages like `libstdc++.so.6:version 'GLIBCXX_3.4.15' not found` during `make mattest`, then it usually means that your Matlab's runtime libraries do not match your compile-time libraries. You may need to do the following before you start Matlab:\r\n\r\n    export LD_LIBRARY_PATH=/opt/intel/mkl/lib/intel64:/usr/local/cuda/lib64\r\n    export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6\r\n\r\nOr the equivalent based on where things are installed on your system, and do `make mattest` again to see if the issue is fixed. Note: this issue is sometimes more complicated since during its startup Matlab may overwrite your `LD_LIBRARY_PATH` environment variable. You can run `!ldd ./matlab/+caffe/private/caffe_.mexa64` (the mex extension may differ on your system) in Matlab to see its runtime libraries, and preload your compile-time libraries by exporting them to your `LD_PRELOAD` environment variable.\r\n\r\nAfter successful building and testing, add this package to Matlab search PATH by starting `matlab` from caffe root folder and running the following commands in Matlab command window.\r\n\r\n    addpath ./matlab\r\n\r\nYou can save your Matlab search PATH by running `savepath` so that you don't have to run the command above again every time you use MatCaffe.\r\n\r\n### Use MatCaffe\r\n\r\nMatCaffe is very similar to PyCaffe in usage.\r\n\r\nExamples below shows detailed usages and assumes you have downloaded BVLC CaffeNet from [Model Zoo](http://caffe.berkeleyvision.org/model_zoo.html) and started `matlab` from caffe root folder.\r\n\r\n    model = './models/bvlc_reference_caffenet/deploy.prototxt';\r\n    weights = './models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel';\r\n\r\n#### Set mode and device\r\n\r\n**Mode and device should always be set BEFORE you create a net or a solver.**\r\n\r\nUse CPU:\r\n\r\n    caffe.set_mode_cpu();\r\n\r\nUse GPU and specify its gpu_id:\r\n\r\n    caffe.set_mode_gpu();\r\n    caffe.set_device(gpu_id);\r\n\r\n#### Create a network and access its layers and blobs\r\n\r\nCreate a network:\r\n\r\n    net = caffe.Net(model, weights, 'test'); % create net and load weights\r\n\r\nOr\r\n\r\n    net = caffe.Net(model, 'test'); % create net but not load weights\r\n    net.copy_from(weights); % load weights\r\n\r\nwhich creates `net` object as\r\n\r\n      Net with properties:\r\n\r\n               layer_vec: [1x23 caffe.Layer]\r\n                blob_vec: [1x15 caffe.Blob]\r\n                  inputs: {'data'}\r\n                 outputs: {'prob'}\r\n        name2layer_index: [23x1 containers.Map]\r\n         name2blob_index: [15x1 containers.Map]\r\n             layer_names: {23x1 cell}\r\n              blob_names: {15x1 cell}\r\n\r\nThe two `containers.Map` objects are useful to find the index of a layer or a blob by its name.\r\n\r\nYou have access to every blob in this network. To fill blob 'data' with all ones:\r\n\r\n    net.blobs('data').set_data(ones(net.blobs('data').shape));\r\n\r\nTo multiply all values in blob 'data' by 10:\r\n\r\n    net.blobs('data').set_data(net.blobs('data').get_data() * 10);\r\n\r\n**Be aware that since Matlab is 1-indexed and column-major, the usual 4 blob dimensions in Matlab are `[width, height, channels, num]`, and `width` is the fastest dimension. Also be aware that images are in BGR channels.** Also, Caffe uses single-precision float data. If your data is not single, `set_data` will automatically convert it to single.\r\n\r\nYou also have access to every layer, so you can do network surgery. For example, to multiply conv1 parameters by 10:\r\n\r\n    net.params('conv1', 1).set_data(net.params('conv1', 1).get_data() * 10); % set weights\r\n    net.params('conv1', 2).set_data(net.params('conv1', 2).get_data() * 10); % set bias\r\n\r\nAlternatively, you can use\r\n\r\n    net.layers('conv1').params(1).set_data(net.layers('conv1').params(1).get_data() * 10);\r\n    net.layers('conv1').params(2).set_data(net.layers('conv1').params(2).get_data() * 10);\r\n\r\nTo save the network you just modified:\r\n\r\n    net.save('my_net.caffemodel');\r\n\r\nTo get a layer's type (string):\r\n\r\n    layer_type = net.layers('conv1').type;\r\n\r\n#### Forward and backward\r\n\r\nForward pass can be done using `net.forward` or `net.forward_prefilled`. Function `net.forward` takes in a cell array of N-D arrays containing data of input blob(s) and outputs a cell array containing data from output blob(s). Function `net.forward_prefilled` uses existing data in input blob(s) during forward pass, takes no input and produces no output. After creating some data for input blobs like `data = rand(net.blobs('data').shape);` you can run\r\n    \r\n    res = net.forward({data});\r\n    prob = res{1};\r\n\r\nOr\r\n\r\n    net.blobs('data').set_data(data);\r\n    net.forward_prefilled();\r\n    prob = net.blobs('prob').get_data();\r\n\r\nBackward is similar using `net.backward` or `net.backward_prefilled` and replacing `get_data` and `set_data` with `get_diff` and `set_diff`. After creating some gradients for output blobs like `prob_diff = rand(net.blobs('prob').shape);` you can run\r\n\r\n    res = net.backward({prob_diff});\r\n    data_diff = res{1};\r\n\r\nOr\r\n\r\n    net.blobs('prob').set_diff(prob_diff);\r\n    net.backward_prefilled();\r\n    data_diff = net.blobs('data').get_diff();\r\n    \r\n**However, the backward computation above doesn't get correct results, because Caffe decides that the network does not need backward computation. To get correct backward results, you need to set `'force_backward: true'` in your network prototxt.**\r\n\r\nAfter performing forward or backward pass, you can also get the data or diff in internal blobs. For example, to extract pool5 features after forward pass:\r\n\r\n    pool5_feat = net.blobs('pool5').get_data();\r\n\r\n#### Reshape\r\n\r\nAssume you want to run 1 image at a time instead of 10:\r\n\r\n    net.blobs('data').reshape([227 227 3 1]); % reshape blob 'data'\r\n    net.reshape();\r\n\r\nThen the whole network is reshaped, and now `net.blobs('prob').shape` should be `[1000 1]`;\r\n\r\n#### Training\r\n\r\nAssume you have created training and validation lmdbs following our [ImageNET Tutorial](http://caffe.berkeleyvision.org/gathered/examples/imagenet.html), to create a solver and train on ILSVRC 2012 classification dataset:\r\n\r\n    solver = caffe.Solver('./models/bvlc_reference_caffenet/solver.prototxt');\r\n\r\nwhich creates `solver` object as\r\n\r\n      Solver with properties:\r\n\r\n              net: [1x1 caffe.Net]\r\n        test_nets: [1x1 caffe.Net]\r\n\r\nTo train:\r\n\r\n    solver.solve();\r\n\r\nOr train for only 1000 iterations (so that you can do something to its net before training more iterations)\r\n\r\n    solver.step(1000);\r\n\r\nTo get iteration number:\r\n\r\n    iter = solver.iter();\r\n\r\nTo get its network:\r\n\r\n    train_net = solver.net;\r\n    test_net = solver.test_nets(1);\r\n\r\nTo resume from a snapshot \"your_snapshot.solverstate\":\r\n\r\n    solver.restore('your_snapshot.solverstate');\r\n\r\n#### Input and output\r\n\r\n`caffe.io` class provides basic input functions `load_image` and `read_mean`. For example, to read ILSVRC 2012 mean file (assume you have downloaded imagenet example auxiliary files by running `./data/ilsvrc12/get_ilsvrc_aux.sh`):\r\n\r\n    mean_data = caffe.io.read_mean('./data/ilsvrc12/imagenet_mean.binaryproto');\r\n\r\nTo read Caffe's example image and resize to `[width, height]` and suppose we want `width = 256; height = 256;`\r\n\r\n    im_data = caffe.io.load_image('./examples/images/cat.jpg');\r\n    im_data = imresize(im_data, [width, height]); % resize using Matlab's imresize\r\n\r\n**Keep in mind that `width` is the fastest dimension and channels are BGR, which is different from the usual way that Matlab stores an image.** If you don't want to use `caffe.io.load_image` and prefer to load an image by yourself, you can do\r\n\r\n    im_data = imread('./examples/images/cat.jpg'); % read image\r\n    im_data = im_data(:, :, [3, 2, 1]); % convert from RGB to BGR\r\n    im_data = permute(im_data, [2, 1, 3]); % permute width and height\r\n    im_data = single(im_data); % convert to single precision\r\n\r\nAlso, you may take a look at caffe/matlab/demo/classification_demo.m to see how to prepare input by taking crops from an image.\r\n\r\nWe show in caffe/matlab/hdf5creation how to read and write HDF5 data with Matlab. We do not provide extra functions for data output as Matlab itself is already quite powerful in output.\r\n\r\n#### Clear nets and solvers\r\n\r\nCall `caffe.reset_all()` to clear all solvers and stand-alone nets you have created."},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/106979632","html_url":"https://github.com/BVLC/caffe/pull/2505#issuecomment-106979632","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/2505","id":106979632,"user":{"login":"shelhamer","id":311113,"avatar_url":"https://avatars.githubusercontent.com/u/311113?v=3","gravatar_id":"","url":"https://api.github.com/users/shelhamer","html_url":"https://github.com/shelhamer","followers_url":"https://api.github.com/users/shelhamer/followers","following_url":"https://api.github.com/users/shelhamer/following{/other_user}","gists_url":"https://api.github.com/users/shelhamer/gists{/gist_id}","starred_url":"https://api.github.com/users/shelhamer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shelhamer/subscriptions","organizations_url":"https://api.github.com/users/shelhamer/orgs","repos_url":"https://api.github.com/users/shelhamer/repos","events_url":"https://api.github.com/users/shelhamer/events{/privacy}","received_events_url":"https://api.github.com/users/shelhamer/received_events","type":"User","site_admin":false},"created_at":"2015-05-30T03:08:09Z","updated_at":"2015-05-30T03:08:09Z","body":"Yeah, leave it to MATLAB to have the One True Plus."}}