Getting Data In


Monitor files and directories



Monitor files and directories

Splunk has two file input processors: monitor and upload. For the most part, you can use monitor to add all your data sources from files and directories. However, you might want to use upload to add one-time inputs, such as an archive of historical data.


How monitor works in Splunk

Specify a path to a file or directory and Splunk's monitor processor consumes any new data written to that file or directory. This is how you can monitor live application logs such as those coming from J2EE or .NET applications, Web access logs, and so on. Splunk continues to monitor and index the file or directory as new data appears. You can also specify a mounted or shared directory, including network file systems, so long as Splunk can read from the directory. If the specified directory contains subdirectories, Splunk recursively examines them for new files.

Splunk checks for the file or directory specified in a monitor configuration on Splunk start and restart. If the file or directory is not present on start, Splunk checks for it again in 24 hour intervals from the time of the last restart. Subdirectories of monitored directories are scanned continuously. To add new inputs without restarting Splunk, use Splunk Web or the CLI. If you want Splunk to find potential new inputs automatically, use crawl.

When using monitor, note the following:

  • On most file systems, files can be read even as they are being written to. However, Windows file systems have the ability to prevent files from being read while they are being written, and some Windows programs may use these modes, though most do not.
  • Upon restart, Splunk continues processing files where it left off.
  • Splunk decompresses archive files before it indexes them. It can handle these common archive file types: .tar, .gz, .bz2, .tar.bz2 , and .zip.

Note: You cannot currently use both monitor and file system change monitor to follow the same directory or file. If you want to see changes in a directory, use file system change monitor. If you want to index new events in a directory, use monitor.

Note: Monitor inputs should not overlap. That is, monitoring /a/path while also monitoring /a/path/subdir will produce unreliable results. Similarly, monitor inputs that watch the same directory with different whitelists, blacklists, and wildcard components are not supported.

Why use upload or batch



Use the batch input type in inputs.conf to load files once and destructively. By default, Splunk's batch processor is located in $SPLUNK_HOME/var/spool/splunk. If you move a file into this directory, Splunk indexes it and then deletes it.


This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 View the Article History for its revisions.


When using ../var/spool/splunk and setting "segment in path" and setting the "segment number" of the host, the directory is not deleted, either is the file within.

August 10, 2011


Was this documentation topic helpful?



Feedback submitted, thanks!