{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511/events","html_url":"https://github.com/tensorflow/tensorflow/issues/511","id":122146339,"number":511,"title":"Large RNN graphs, incredibly slow graph creation and step time...","user":{"login":"wchan","id":1131892,"avatar_url":"https://avatars.githubusercontent.com/u/1131892?v=3","gravatar_id":"","url":"https://api.github.com/users/wchan","html_url":"https://github.com/wchan","followers_url":"https://api.github.com/users/wchan/followers","following_url":"https://api.github.com/users/wchan/following{/other_user}","gists_url":"https://api.github.com/users/wchan/gists{/gist_id}","starred_url":"https://api.github.com/users/wchan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wchan/subscriptions","organizations_url":"https://api.github.com/users/wchan/orgs","repos_url":"https://api.github.com/users/wchan/repos","events_url":"https://api.github.com/users/wchan/events{/privacy}","received_events_url":"https://api.github.com/users/wchan/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"ludimagister","id":15710643,"avatar_url":"https://avatars.githubusercontent.com/u/15710643?v=3","gravatar_id":"","url":"https://api.github.com/users/ludimagister","html_url":"https://github.com/ludimagister","followers_url":"https://api.github.com/users/ludimagister/followers","following_url":"https://api.github.com/users/ludimagister/following{/other_user}","gists_url":"https://api.github.com/users/ludimagister/gists{/gist_id}","starred_url":"https://api.github.com/users/ludimagister/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ludimagister/subscriptions","organizations_url":"https://api.github.com/users/ludimagister/orgs","repos_url":"https://api.github.com/users/ludimagister/repos","events_url":"https://api.github.com/users/ludimagister/events{/privacy}","received_events_url":"https://api.github.com/users/ludimagister/received_events","type":"User","site_admin":false},"milestone":null,"comments":15,"created_at":"2015-12-14T22:28:03Z","updated_at":"2015-12-17T04:36:05Z","closed_at":null,"body":"When I build a large seq2seq RNN graph (i.e., 3000 timesteps encoder, and 100 time steps decoder), both the graph construction and step time is insanely slow.\r\n\r\nFor example, using approx 3000 timesteps of stacked 4 layers of GRUs (in a seq2seq model w/ attention), we get something like this:\r\n\r\n*** graph creation time ***\r\ncreate_encoder graph time 77.442740\r\ncreate_decoder graph time 7.298214\r\ncreate_loss graph time 0.934293\r\ncreate_optimizer graph time 349.426908\r\ncreate_model graph time 489.119399\r\n\r\ncreate_optimizer is the part where i call something like this (pulled from the tutorial):\r\n  def create_optimizer(self):\r\n    start_time = time.time()\r\n\r\n    params = tf.trainable_variables()\r\n\r\n    self.gradient_norms = []\r\n    self.updates = []\r\n    opt = tf.train.GradientDescentOptimizer(0.001)\r\n\r\n    gradients = tf.gradients(self.losses, params)\r\n    clipped_gradients, norm = tf.clip_by_global_norm(gradients, self.max_gradient_norm)\r\n\r\n    self.gradient_norms.append(norm)\r\n    self.updates.append(opt.apply_gradients(\r\n        zip(clipped_gradients, params), global_step=self.global_step))\r\n\r\n    print('create_optimizer graph time %f' % (time.time() - start_time))\r\n\r\n*** step time ***\r\nstep_time: 142.356251001\r\n\r\nWhen I run nvidia-smi while the graph is computing , the utilization is something like 12% ... definitely something wrong...\r\n\r\nIn my personal framework (not TF), creating a graph of equal size takes approx 2 secs (as opposed to TF almost 8 mins) and the step time is less 4 seconds (as opposed to TF 2 mins).... something really weird is going on.\r\n\r\nI wasn't able to profile / debug it too much (are there public tools for that? i cant find anything on tensorboard). also, is there a command i can run to confirm how many weights i have (and the dimensions and the device location of the variables).\r\n\r\nI've confirmed with the tensorboard and log_device_placement=True, and seems like most of the graph is in the GPU (except for embedding layers).\r\n\r\nIf you want a link to my graph, send me an email.\r\n\r\nThanks All!"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/165334893","html_url":"https://github.com/tensorflow/tensorflow/issues/511#issuecomment-165334893","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511","id":165334893,"user":{"login":"wchan","id":1131892,"avatar_url":"https://avatars.githubusercontent.com/u/1131892?v=3","gravatar_id":"","url":"https://api.github.com/users/wchan","html_url":"https://github.com/wchan","followers_url":"https://api.github.com/users/wchan/followers","following_url":"https://api.github.com/users/wchan/following{/other_user}","gists_url":"https://api.github.com/users/wchan/gists{/gist_id}","starred_url":"https://api.github.com/users/wchan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wchan/subscriptions","organizations_url":"https://api.github.com/users/wchan/orgs","repos_url":"https://api.github.com/users/wchan/repos","events_url":"https://api.github.com/users/wchan/events{/privacy}","received_events_url":"https://api.github.com/users/wchan/received_events","type":"User","site_admin":false},"created_at":"2015-12-17T04:36:05Z","updated_at":"2015-12-17T04:36:05Z","body":"i've coded up a monolithic gru op, it compiles but im getting a really weird dynamic link error when I do the python import statement for just \"import tensorflow as tf\":\r\n\r\nImportError: /home/wchan/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so: undefined symbol: _ZN5Eigen8internal14TensorExecutorIKNS_14TensorAssignOpINS_9TensorMapINS_6TensorIfLm1ELi1ElEELi1EEEKNS_19TensorCwiseBinaryOpINS0_20scalar_difference_opIKfEEKNS7_INS0_17scalar_product_opIS9_S9_EEKNS3_INS4_IS9_Lm1ELi1ElEELi1EEESF_EESH_EEEENS_9GpuDeviceELb1ELb0EE3runERSL_RKSM_\r\n\r\nAnyone see something like that before? I'm guessing this must be related to the Eigen::Tensor?\r\n\r\nimpl path: https://github.com/wchan/tensorflow/blob/master/tensorflow/core/kernels/gru_op.cc"}}