Data analysis of log files – How to find a pattern?
<p>My company has slightly more than 300 vehicle based windows CE 5.0 mobile devices that all share the same software and usage model of Direct Store Delivery during the day then doing a Tcom at the home base every night.  There is an unknown event(s) that results in the device freaking out and rebooting itself in the middle of the day.  Frequency of this issue is ~10 times per week across the fleet of computers that all reboot daily, 6 days a week.  The math is 300*6=1800 boots per week (at least) 10/1800= 0.5%.  I realize that number is very low, but it is more than my boss wants to have. </p>

<p>My challenge, is to find a way to scan through several thousand logfille.txt files and try to find some sort of pattern.  I KNOW there is a pattern here somewhere. I’ve got a couple ideas of where to start, but I wanted to throw this out to the community and see what suggestions you all might have.</p>

<p>A bit of background on this issue.  The application starts a new log file at each boot.  In an orderly (control) log file, you see the app startup, do its thing all day, and then start a shutdown process in a somewhat orderly fashion 8-10 hours later.  In a problem log file, you see the device startup and then the log ends without any shutdown sequence at all in a time less than 8 hours.  It then starts a new log file which shares the same date as the logfile1.old that it made in the rename process.  The application that we have was home grown by windows developers that are no longer with the company.  Even better, they don’t currently know who has the source at the moment.</p>

<p>I’m aware of the various CE tools that can be used to detect memory leaks (DevHealth, retail messages, etc..) and we are investigating that route as well, however I’m convinced that there is a pattern to be found, that I’m just not smart enough to find. There has to be a way to do this using Perl or Python that I’m just not seeing. Here are two ideas I have.</p>

<p>Idea 1 –  Look for trends in word usage.
Create an array of every unique word used in the entire log file and output a count of each word.  Once I had a count of the words that were being used, I could run some stats on them and look for the non-normal events.  Perhaps the word “purple” is being used 500 times in a 1000 line log file ( there might be some math there?) on a control and only 4 times on a 500 line problem log?  Perhaps there is a unique word that is only seen in the problem files.  Maybe I could get a reverse “word cloud”? </p>

<p>Idea 2 – categorize lines into entry-type and then look for trends in the sequence of type of entry type?
The logfiles already have a predictable schema that looks like this =  Level|date|time|system|source|message<br>
I’m 99% sure there is a visible pattern here that I just can’t find.  All of the logs got turned up to “super duper verbose” so there is a boatload of fluff (25 logs p/sec , 40k lines per file) that makes this even more challenging.  If there isn’t a unique word,  then this has almost got to be true.  How do I do this?</p>

<p>Item 3 – Hire a windows CE platform developer
Yes, we are going down that path as well, but I KNOW there is a pattern I’m missing.  They will use the tools that I don’t have) or make the tools that we need to figure out what’s up.  I suspect that there might be a memory leak, radio event or other event that platform tools I’m sure will show.</p>

<p>Item 4 – Something I’m not even thinking of that you have used.
There have got to be tools out there that do this that aren’t as prestigious as a well-executed python script, and I’m willing to go down that path, I just don’t know what those tools are.</p>

<p>Oh yeah, I can’t post log files to the web, so don’t ask.  The users are promising to report trends when they see them, but I’m not exactly hopeful on that front.  All I need to find is either a pattern in the logs, or steps to duplicate</p>

<p>So there you have it.  What tools or techniques can I use to even start on this?  </p>