Is More Data Always Better?

google think magazine data overload obesity information ideas processing analysisThere has been a discovery in the online marketing and data/statistics world in the last few years. We have had more websites, products and tools created online than we can possibly keep track of. The terms to describe this deluge of activity we have been hearing the most are “data overload” and “information overload” from both companies and consumers. This Google Magazine uses the term Data Obesity to describe this phenomenon.

They ask the question, why is more data always better?

I think the idea of “more data us better” is common from people who lived before the Internet was prevalent. We had to work hard to find data. Researching something meant going to a library and looking in a card catalog (or maybe something called Gopher) and then finding your way around the Dewey decimal system to find that book. And then sometimes they didn’t even have the book because it was checked out or possibly it was just filed wrong because nobody understood the Dewey decimal system.

On a related note recently we got invited to my cousin’s wedding in Santa Fe New Mexico. My dad promptly went to the library and checked out 3 books on Santa Fe and New Mexico. I cringed. He asked how to find out the flights to book something without a travel agent. I realized I have been traveling since 2000 this way and he stopped traveling about that time so he never has. I introduced him to Travelocity, it was mind blowing and a bit of data overload compared with the OAG book he used to use in the 80’s.

The point here is that finding data was really difficult. People had control over its distribution because it was in print. When it became more freely accessible due to Google and other companies efforts we assumed this would be good, because people could remember where to find it and use it whenever we wanted. We never thought it would get this big so fast. Now travel sites are overwhelming, they have too many choices and there are too many of them trying to get you to opt into something you don’t want while being over charged for bringing a suitcase on a flight. This is just one example of how data has gone exponential so quickly.

Others of us have come to a data overload conclusion when they have 200 emails in several in-boxes, 1000+ rss reader posts from feeds waiting, several work projects, 500+ Facebook wall posts in their feed and hundreds of tweets that have gone un-read. This is among a climate where you have to follow-up with projects 5-10 times to get things done, post blogs/tweets/FB status updates daily to keep on people’s radar, empty the DVR so it doesn’t get overloaded and auto delete something you really wanted, listen to the radio on the way to work just in case something big happens and still find time to scoop the litter box before it gets full and the cats poop on the floor.

And the real purpose in all those tweets/FB posts and feeds is that you business changes yearly and if you don’t know about the latest trend and some real insights about it before your boss asks about it, you won’t have a job for all that long. (in digital marketing)

Having data overload be a “good” problem to have from some people’s perspective (as in that it is growth oriented). The democratization of publishing combined with tracking methodology and databases have all contributed to this problem, giving everyone a voice, a potential following of readers, a data trail to analyze and method to say something important online 24/7/365.  And then we have an even bigger problem of processing what is being said, figuring out if it is important or not and sharing/processing/saving it in some way if it is. Acting on that data is way down the line and many of us don’t even get there.

And this isn’t even the big problem with data overload. Where will we store it all? Why do tweets disappear from search so quickly? Because there are millions of them and the failwhale is full. According to the ThinkQuarterly UK, there are 800 Exabytes of data/information created every two days. It took humans from the beginning of civilization until 2003 to create the first 800 Exabytes, and we’re on a roll now.

Where does all this seemingly random data go? How will we know what it says without having to go into a database table and read specific field information? Where are the software tools to manage all this and still give humans the ability to customize the out put in ways that match the behavior or business purposes that we really need? Does any of this stuff ever get deleted?

These are all huge questions we have to answer as more people publish, share, create, track and do business online. We also have to weigh the possibilities of sharing data openly and locking it behind walls as well as how will people comprehensively find what they need when they want to as well as gauge the validity/accuracy of the information presented?

I’m betting on paid services for personal and business data management/archiving & Analysis tools. We will pay for good analysis, good data access & processing and good reliability/backups when we feel the pain of missing good insight, losing good data and just too much happening. Both personally and professionally. But unless you know how to work with SAP, SPSS, SQL, Oracle or a bunch of other systems data management is largely out of your control at this point. They are the librarians of our digital data and they need to find a workable way to Dewey decimal system it back into order and allow us to use it as humans need to.

The Negatives of Social Networking Media

All the world is a Buzz about Facebook & Twitter these days. It’s almost like MySpace circa 2007, Google circa 2003 or Microsoft circa 1998. I don’t doubt the success, innovation or long-term viability of these social networking sites but I have seen that there are flaws in the system that mean that things won’t be perfect with the business along the way and we’re in for a bumpy road. Basically my point is that for all these sites give us in entertainment, social connections and opportunity they also have some negatives that are almost the equal and opposite pendulum action.

1. Time Suck – all social networking sites are using your time that you used to devote to other things. Maybe in some cases this is actually a better use of your time (instead of TV) but in most cases its time spent that you used to use for researching new information for work projects,  time actually spent talking with people in person (family/friends) or time spent doing things that really need to be done at work or home. Once the brain gets trained that you can go socialize instead of work at those times of day it’s a habit extremely hard to break. For all of us procrastinators looking for instant gratification its a real problem keeping up with work and affects the overall productivity of companies and the country as a whole. Internet access is much more prevalent and has far more users during the business day than it does at night, so there’s the proof. Unless your job is trolling these sites for sales prospects by “connecting” and making “relationships” with your customers, its a waste of time to spend more than 15 min a day.

2. Privacy – Of all the details analyzed about consumer privacy online (on Facebook) in the last few weeks the most suprising thing I’ve seen is that people really don’t care about their information online. Sure, nobody is going to post a ss number or cc number on their profile (duh) but they don’t really seem to realize the power of logging all their social interactions in one database and selling access to retailers and cpg companies who have even larger databases of information to analyze and strategize with. Is it really as fun when most of your friends are companies selling you things all the time? Twitter already has morphed into the largest opt in direct marketing platform I’ve ever seen. If people keep using it at this rate it will surpass email. The other obvious issues come with the work life balance thing and when people friend work makes and think nobody will see them rant about work or post drunk pictures on a sick day, but then again I’ve heard that its just people naturally selecting themselves out of the working pool.

3. Logic – the other issues I’ve seen coming for a while have to do with how everything that is built from large databases online with lots of consumer data seems to not work properly. There is always some algorithm developed by a science tech guy based on some theoretical calculus and it doesn’t provide relevant results. Which brings me to a repeating theme of data right now: we don’t really know what to do with it yet. Nobody knows enough real info about their customers to target them. (who has a budget for that?) And the database people just like to say they improved things a statistically insignificant amount with an algorithm tweak. The marketing strategy/process should always start with offline real life information about people and products and then develop an algorithm to show you information in that way. I don’t know why it’s always done backwards but it will keep our results irrelevant and marketing dollars wasted for a long time to come.

Why I dislike Large Blogs

I love blogs. I have been blogging since 2002 when my friend Mugsy emailed me and told me to sign up for LiveJournal. A lot has changed about blogging since then, but the revolutionary idea that if you can type, you can publish easily in a word-processing-like interface on the internet has not. The method of blogging to share knowledge by and for non-programming type people is still spreading to the corners of the globe and helping people’s voices be heard in ways we never thought possible.

At the same time I am growing more frustrated with the technorati and the overload of emails, posts, rss feeds and spam arriving on my accounts daily. I am trying my best to stay on top of the active topics in  the  user generated content world as it has forked into many roads that include blogging, social networking, social ads, microblogging and a whole host of a million little startups with other concepts they want to share with the world. (more than can be kept up with or can survive even if they do all innovate)

I have had to scale back my online content consumption several times over the years when it was in danger of taking over my life and all my time. But lately this getting married thing has taken a large chunk of time out of my life too, (even after the wedding) and as a result I am trying to glean all my updates and news knowledge into smaller and smaller bits of time. (apparently being married means I have to do work around the house and spend a lot of time trying to motivate my husband to stop watching hours of TV and do things around the house. Life just got more complicated and we have to learn how to cook, fix things, do laundry and empty the cat-poop-box with much larger quantities now). My work is also very busy (analytics and metrics seem to go nuts in recessions) and no spare time is to be had anywhere in the schedule.

Therefore, I have gone through many iterations of un-subscribe weeks in my email boxes and cut back drastically on email newsletters, of which once I found very enlightening. Most marketing/advertising/analytics/metrics/SEO/SEM email newsletters  these days aren’t as willing to share any real actionable info without you spending a lot of $ so out they go.

I tried to update myself by trying an RSS reader again (3rd try) and I think its been a few months but I am overwhelmed by that too. Its way to easy to get more than 1,000 unread items in the reader and when it doesn’t tell me the exact number anymore I am less motivated to tackle it because it seems impossible.

I have found Google Reader to be good for sunday afternoon fun feed reading and more personal fun  topics/blogs though. Home design is a great topic in the reader since you really have to see it all to learn.

On the other hand I am re-subscribing to some email newsletters and just un-subscribing altogether to others who insist on posting 30-50 items per day! (assholes!) How is one person supposed to read that many posts per blog per day? It’s impossible and on some level, rude.

I know why they do this. It is partially a play to keep new items being published every few hours to keep the Internet addicts coming back for more traffic and it is also a play for search engine dominance by having more content in the engine for every possible term than anyone else. These teams of writers churn out mostly regurgitated posts about content repurposed from other blogs without much new insight. Some do deliver genuine news and content you can use but scanning through 50 posts is way slower than scanning 5 emails. The content and pics seem to load soooo sloooowly and an email you read, scan and go to what you want quickly. Big offenders of this are ReadWriteWeb (on volume and not separating feeds), Silicon Valley Insider (regurgitating and trying to predict the future even though they’re usually wrong), SEO Roundtable, Apartment Therapy (OMG, holy re-post everyone elses content and fill up with summary posts daily to waste everyones time, generate page views and sell ads), Jalopnik (jebus stop showing us every detail of the 24 hours of Lemons in every city across the country and asking us what our favorite imaginary dream car in a movie with Bruce Willis: waste of space, use summary feeds please! On a cable bandwidth line it takes forever to load all these damn images!) and Media Post (phhbbtt). ALL THESE BLOGS have been banned from my RSS Reader. Some have been demoted to email updates but others are just gone.

Also, I’m not programmed to think to go see my rss feeds yet either so I often forget about them for several days after a good several hour scanning session finally getting the numbers down to below 200 new items. then I return the next time to see 1,000+ again and feel defeated. In contrast I have OCD about keeping a clean email box, and completely forget about facebook until I am completely bored. I guess that is a sign of my age bracket. (34)

I wish that this spammy fluf put out there to fill space could be eliminated. I also wish that these blogs would split their feeds into sections so you would be able to just get the posts you were interested in. Like if new original content and re-purposed other people’s content were separated in 2 feeds, it would be a big help.

I would also recommend that they stop doing summary posts. They piss me off. I wait a minute or 2 for something to load in the darn reader only to see its the same posts from the local editions of the same blog.  Poo, if that happens 15 times in a day I could have spent that time sleeping and then I’m annoyed. 

These blogs also do this because they are in some get-rich-quick rush to make money as a profitable business before Google figures it out and bans them or something. Yes, blogs have an elitism to them that says, duh, if I can make a slice of the money publishing from what the Tribune used to, I am going to do this as fast and as hard as I can. And it over saturates the web with watered down content that is just filler mostly, even if it does increase ad impressions and some adsense revenue if you’re into web-welfare payments.

I also would like to recommend that if you want to start a blog you keep the posts to no more than 2-3 a day and resist the urge to just regurgitate other people’s posts and link to them saying how great they are. Research things you are really interested in and share your own unique experiences. Any web-bot can be an aggregator, what we need more of is real people sharing experiences and knowledge to make social media stick and not die out because of spam/splogs and info-overload. It is these people who become trusted advisors and get the visitors who come back again and again.

And this is also better for the rest of us who have to go clean the cat-poop-box and have a life offline now that they are married.

Update 10/29/09

I guess Robert Scoble agrees with me to a point, though he blames Google Reader for a bad format and experience and not the blog owners for copious amounts of useless content hiding the good stuff. I guess there is always room for improvement and certain people discover it before others depending on how they use the info/product.

Update 11/5/09

How much content is too much content? Read Write Web chronicles these mega content sites and their race to populate the web all by themselves by posting 200+ posts per day. We should call it the business model.

SEO Update from Chicago









Why the Microsoft & Yahoo Search Deal Sucks








New Media and New Information Paradigms





Is Google Making us Stopid? I think not!

Is Google Making us Stopid? I think not!
















Some things I still do that have survived the internet pruning:


2. I keep up with emails from work and friends.








