{"action":"created","issue":{"url":"https://api.github.com/repos/owncloud/core/issues/15219","labels_url":"https://api.github.com/repos/owncloud/core/issues/15219/labels{/name}","comments_url":"https://api.github.com/repos/owncloud/core/issues/15219/comments","events_url":"https://api.github.com/repos/owncloud/core/issues/15219/events","html_url":"https://github.com/owncloud/core/issues/15219","id":64535076,"number":15219,"title":"high cpu load after upgrade 7.0.4 to 8.0.2 and lucene disabled but still present in the logs","user":{"login":"seal20","id":6326493,"avatar_url":"https://avatars.githubusercontent.com/u/6326493?v=3","gravatar_id":"","url":"https://api.github.com/users/seal20","html_url":"https://github.com/seal20","followers_url":"https://api.github.com/users/seal20/followers","following_url":"https://api.github.com/users/seal20/following{/other_user}","gists_url":"https://api.github.com/users/seal20/gists{/gist_id}","starred_url":"https://api.github.com/users/seal20/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seal20/subscriptions","organizations_url":"https://api.github.com/users/seal20/orgs","repos_url":"https://api.github.com/users/seal20/repos","events_url":"https://api.github.com/users/seal20/events{/privacy}","received_events_url":"https://api.github.com/users/seal20/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/owncloud/core/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/owncloud/core/labels/performance","name":"performance","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-26T14:07:26Z","updated_at":"2015-03-27T13:24:48Z","closed_at":null,"body":"### Steps to reproduce\r\nI don't really know how to reproduce it but it keeps happening... What I suppose\r\n\r\n1. Upgrade to 8.0.2\r\n2. Lucene start indexing or optimizing resulting in very high cpu load\r\n3. Disable lucene\r\n4. the high cpu load keep happening and the log show error related to lucene\r\n\r\n### Expected behaviour\r\nno high cpu load and once lucence is disabled it doesn't do anything\r\n\r\n### Actual behaviour\r\nhigh cpu load and still lucene errors in the log\r\n\r\n### Server configuration\r\n**Operating system**: Centos 6.6 (virtuozzo VPS)\r\n\r\n**Web server:**Apache/2.4.10 (Unix)\r\n\r\n**Database:**MySQL Ver 14.14 Distrib 5.5.42, for Linux (x86_64) using readline 5.1\r\n\r\n**PHP version:**PHP 5.4.35 (cli) (built: Nov 29 2014 08:31:07) \r\nZend Engine v2.4.0, eAccelerator v0.9.6.1-ea, ionCube PHP Loader v4.6.1, Zend Guard Loader v3.3,\r\n\r\n**ownCloud version:** (see ownCloud admin page)\r\n8.0.2.0\r\n\r\n**Updated from an older ownCloud or fresh install:**\r\nyes from 7.0.4\r\n\r\n**List of activated apps:**\r\n\r\n```\r\nEnabled:\r\n - activity\r\n - calendar\r\n - contacts\r\n - documents\r\n - files\r\n - files_pdfviewer\r\n - files_sharing\r\n - files_texteditor\r\n - files_trashbin\r\n - files_versions\r\n - files_videoviewer\r\n - firstrunwizard\r\n - gallery\r\n - news\r\n - ownnote\r\n - tasks\r\n - templateeditor\r\n - updater\r\nDisabled:\r\n - external\r\n - files_encryption\r\n - files_external\r\n - files_locking\r\n - provisioning_api\r\n - search_lucene\r\n - user_external\r\n - user_ldap\r\n - user_webdavauth\r\n```\r\n**The content of config/config.php:**\r\n\r\n```\r\n$CONFIG = array (\r\n  'instanceid' => 'oc3053678514',\r\n  'passwordsalt' => '',\r\n  'datadirectory' => '/path/oc_data',\r\n  'dbtype' => 'mysql',\r\n  'version' => '8.0.2.0',\r\n  'dbname' => '',\r\n  'dbhost' => 'localhost',\r\n  'dbtableprefix' => 'oc_',\r\n  'dbuser' => '',\r\n  'dbpassword' => '',\r\n  'installed' => true,\r\n  'forcessl' => true,\r\n  'loglevel' => 0,\r\n  'log_rotate' => '104857600',\r\n  'theme' => '',\r\n  'maintenance' => false,\r\n  'trusted_domains' => \r\n  array (\r\n    0 => '.com',\r\n  ),\r\n  'mail_smtpmode' => 'smtp',\r\n  'mail_from_address' => 'notifications',\r\n  'mail_domain' => '.com',\r\n  'overwrite.cli.url' => '/owncloud',\r\n  'allow_user_to_change_display_name' => true,\r\n  'logtimezone' => 'Asia/Tokyo',\r\n  'preview_libreoffice_path' => '/usr/bin/libreoffice',\r\n  'preview_office_cl_parameters' => ' --headless --nologo --nofirststartwizard --invisible --norestore -convert-to pdf -outdir ',\r\n  'secret' => ',\r\n  'mail_smtpauthtype' => 'LOGIN',\r\n  'mail_smtpauth' => 1,\r\n  'mail_smtpport' => '465',\r\n  'appcodechecker' => false,\r\n  'custom_csp_policy' => 'default-src \\'self\\'; script-src \\'self\\' \\'unsafe-inline\\' \\'unsafe-eval\\'; style-src \\'self\\' \\'unsafe-inline\\'; frame-src *; img-src *; font-src \\'self\\' data:; media-src *',\r\n); \r\n```\r\nThe last custom_csp_policy is required by ownnotes\r\n\r\n**Are you using external storage, if yes which one:** local/smb/sftp/...\r\nno\r\n\r\n**Are you using encryption:** yes/no\r\nno\r\n\r\n**Are you using an external user-backend, if yes which one:** LDAP/ActiveDirectory/Webdav/...\r\nno\r\n\r\n\r\n### Client configuration\r\nnot relevant as it happens when nobody os connected, perhaps cron.php?\r\n\r\n**Browser:**\r\n\r\n**Operating system:**\r\n\r\n### Logs\r\n#### Web server error log\r\n```\r\n[Thu Mar 26 04:21:43.892165 2015] [authz_core:error] [pid 7066] [client 91.121.100.66:41561] AH01630: client denied by server configuration: /home/agoremix/public_html/.htaccess\r\n[Thu Mar 26 05:00:03.454043 2015] [core:crit] [pid 7065] (13)Permission denied: [client 184.105.139.67:37796] AH00529: /home/agoremix/public_html/oc_data/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/home/agoremix/public_html/oc_data/' is executable\r\n[Thu Mar 26 05:00:03.454104 2015] [authz_core:error] [pid 7065] [client 184.105.139.67:37796] AH01630: client denied by server configuration: /home/agoremix/public_html/.htaccess\r\n[Thu Mar 26 05:08:16.855082 2015] [mpm_prefork:notice] [pid 20558] AH00171: Graceful restart requested, doing restart\r\n[Thu Mar 26 05:08:16.938089 2015] [auth_digest:notice] [pid 20558] AH01757: generating secret for digest authentication ...\r\n[Thu Mar 26 05:08:17.026797 2015] [ssl:warn] [pid 20558] AH01906: server.domain.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)\r\n[Thu Mar 26 05:08:17.030115 2015] [ssl:warn] [pid 20558] AH01906: books.domain.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)\r\n[Thu Mar 26 05:08:17.063531 2015] [mpm_prefork:notice] [pid 20558] AH00163: Apache/2.4.10 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4 configured -- resuming normal operations\r\n[Thu Mar 26 05:08:17.063547 2015] [core:notice] [pid 20558] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'\r\n[Thu Mar 26 07:03:00.624653 2015] [mpm_prefork:notice] [pid 20558] AH00169: caught SIGTERM, shutting down\r\n\r\n```\r\n\r\n#### ownCloud log (data/owncloud.log)\r\n```\r\n{\"reqId\":\"1b4141ed99b35f76483d6381c1312d0b\",\"remoteAddr\":\"\",\"app\":\"PHP\",\"message\":\"Undefined index: REQUEST_URI at \\/home\\/domain\\/public_html\\/owncloud\\/apps\\/calendar\\/appinfo\\/app.php#41\",\"level\":3,\"time\":\"2015-03-26T22:48:13+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"da309fba08bfab2be71b7285a9602f8a\",\"remoteAddr\":\"\",\"app\":\"PHP\",\"message\":\"Undefined index: REQUEST_URI at \\/home\\/domain\\/public_html\\/owncloud\\/apps\\/calendar\\/appinfo\\/app.php#41\",\"level\":3,\"time\":\"2015-03-26T22:48:50+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"PHP\",\"message\":\"Undefined index: REQUEST_URI at \\/home\\/domain\\/public_html\\/owncloud\\/apps\\/calendar\\/appinfo\\/app.php#41\",\"level\":3,\"time\":\"2015-03-26T23:00:02+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"news\",\"message\":\"Configuration value \\\"simplePieCacheDuration\\\" does not exist. Ignored value.\",\"level\":2,\"time\":\"2015-03-26T23:00:02+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"search_lucene\",\"message\":\"background job optimizing index for sylvain\",\"level\":0,\"time\":\"2015-03-26T23:00:22+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"search_lucene\",\"message\":\"removed 0 files from index\",\"level\":0,\"time\":\"2015-03-26T23:00:22+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"search_lucene\",\"message\":\"background job optimizing index for roger\",\"level\":0,\"time\":\"2015-03-26T23:00:22+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"search_lucene\",\"message\":\"removed 0 files from index\",\"level\":0,\"time\":\"2015-03-26T23:00:22+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"search_lucene\",\"message\":\"background job optimizing index for chouette\",\"level\":0,\"time\":\"2015-03-26T23:00:22+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n{\"reqId\":\"458ec71b2ff62ca44657fdfc03327de2\",\"remoteAddr\":\"\",\"app\":\"search_lucene\",\"message\":\"removed 0 files from index\",\"level\":0,\"time\":\"2015-03-26T23:00:22+09:00\",\"method\":\"--\",\"url\":\"--\"}\r\n```\r\nProblem I don't know exactely at what time the high load happened but I received a high load abuse warning everyday for the last three days and today was the LAST warning... but I couldn't witness myself the high load. If I run the cron job with lucene, there was a high load so I disabled lucene and the cron.php soon finish without high load. However the log still show lucene error and sometimes in the day the high load happens again.\r\nI disabled apache and cron jobs for the moment but I want to find a way to debug this without getting kicked out by my provider.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/owncloud/core/issues/comments/86938940","html_url":"https://github.com/owncloud/core/issues/15219#issuecomment-86938940","issue_url":"https://api.github.com/repos/owncloud/core/issues/15219","id":86938940,"user":{"login":"seal20","id":6326493,"avatar_url":"https://avatars.githubusercontent.com/u/6326493?v=3","gravatar_id":"","url":"https://api.github.com/users/seal20","html_url":"https://github.com/seal20","followers_url":"https://api.github.com/users/seal20/followers","following_url":"https://api.github.com/users/seal20/following{/other_user}","gists_url":"https://api.github.com/users/seal20/gists{/gist_id}","starred_url":"https://api.github.com/users/seal20/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seal20/subscriptions","organizations_url":"https://api.github.com/users/seal20/orgs","repos_url":"https://api.github.com/users/seal20/repos","events_url":"https://api.github.com/users/seal20/events{/privacy}","received_events_url":"https://api.github.com/users/seal20/received_events","type":"User","site_admin":false},"created_at":"2015-03-27T13:24:48Z","updated_at":"2015-03-27T13:24:48Z","body":"I could \"fix\" the high load issue... by disabling some other non-owncloud background jobs.\r\nHowever, I can confirm that even if lucene is disabled it still log errors and lucene indexing is never ending when enabled using a lot of cpu for a very long time. The \"count down of files remaining\" starts at around 140K, goes fast in the begining and stall at 139K after 1 hour, then I killed the php and disabled lucene.\r\nSo I would like to separate this issue in 2, please advise if they better fit in the lucene repo:\r\n\r\n1. Lucene even if disabled still logs error so is still doing something\r\n2. Indexing in lucene use too much cpu and takes too long (duplicate of https://github.com/owncloud/search_lucene/issues/44 ? ; definitevely a lucene issue)"}}