Tuesday, June 25, 2013

Why the gap between the rich and poor will grow, not shrink, and why we eventually need a different economic model

A lot has been said about the income gap in recent years, while anger has grown towards the 1% as more and more people feel their dreams of wealth slip away.   Why is that?    Is it really just greedy corporations and bad government?   I think other forces are equally or more responsible.

20th Century Automation
In the 1920's mechanized equipment led to a massive increase in the productivity per farmer, and as a result, there were far fewer farmers.   In the 1950's auto workers went on strike when auto companies wanted to use robots to build cars.    Automation in all sectors of the manufacturing industry has caused there to be far fewer manufacturing jobs.    In the 20th century people were alarmed about automation, but the sky did not fall.   Increases in efficiency created more wealth for nearly everyone, and there were still plenty of other jobs for displaced labor to migrate to.   There was individual pain in transition, but as a whole the economy continued to grow and there were plenty of jobs.  

21st Century "Intelligent" Automation
The 21st century has seen an acceleration of technology driven by computers and the Internet.    Computers are more capable and the Internet has connected everyone in ways never before possible.   The combination of these two things has led to the automation of what was previously middle class white collar jobs.    Remember how we would go see a travel agent when we wanted a plane ticket?   Almost all tickets are now sold online.    Remember all those telephone operators?    Phone support numbers are way down as automated systems and the Web have brought new ways to solve tech support issues and disseminate information.

Let's think about some future areas:
  • Most retail sales people - Replaced by online retailers like Amazon
  • Drivers  - Replaced by self driving cars 
  • Many teachers - Replaced through online and computer learning techniques
  • Lawyers - Ha, just kidding on that one, although there is some increase in efficiency with sites like LegalZoom.com
Even if an industry isn't transformed, the addition of computers usually increases productivity and reduces the number of workers that are needed.

Is that bad?  Isn't efficiency good?
Well it depends.    More efficiency is almost always good, we get more for less, things get cheaper, higher quality, and we don't need to work as hard.    Efficiency is bad when it can't be integrated into the economy properly.    Or to put it another way, if our economy can't adapt it will suffer and possibly die, just like an animal species that can't adapt quickly enough to its changing surroundings.

Who is this effecting?
The specialized labor force is benefiting.    These are people with unique skills that can't be replaced by automation.    Doctors, Lawyers, Computer Programers, Authors, Artists, etc..   The common theme is that these are well educated specialized knowledge workers.      The salary trend for workers over the the last 50 years shows a massive gap that is widening between the highly educated workforce and the uneducated workforce.      In other words, if you don't have a specialized skill there are very few good jobs for you any more.   

Computerized Automation has taken a bunch of previously good jobs
As more and more middle class jobs are replaced by computers, these workers are being pushed down into the low skill job market.   As more people enter the low skill job market the wages fall due to simple supply and demand economics.  

What can we do?
The obvious answer is to transform more people into highly educated skilled workers, unfortunately that takes a lot of time and a cultural willingness to get there.     There are other ways creative ways that economists and Sci-fi writers have contemplated over the years.
  • Restrict working hours to reduce productivity per worker.    This is what France and Germany have been doing lately.    This artificially reverses the efficiency gains and brings the market back to a temporary equilibrium.    Unfortunately the rest of the world keeps moving forward and this causes the local economy to suffer.
  • Create make work jobs to employ people and give them something to do.    This is what the US did during the great depression.   A variation of this is to prohibit automation in certain jobs to protect the job.    Oregon and New Jersey did this with gas station attendants, it is illegal to self fill, but there are lots of jobs for gas station attendants.   Taken to its logical extreme this becomes something like the Amish, forsaking all modern tools, but having plenty of jobs.
  • Figure out a new status quo for employment.    Does everyone need to work?    
What's this crazy idea about not working?    Isn't that un-American?
As a thought experiment let's imagine a not too distant future where computers have gotten really smart, we have plenty of electrical energy, and robotics have advanced substantially.     Manufacturing jobs will be gone completely.   Either a robot or a 3d printer will make all of the normal products that we use.     Farms will have robotic tractors and robotic cow milking machines.   Our house cleaning will be done by super advanced Roomba's and our cars can automatically drive us any where we want.    A computer will take our order at McDonald's and our food will be automatically prepared and delivered to our table.  I know this sounds like a bad 1960's Sci-fi film, but something similar will be here eventually.  Maybe it will take a really long time, but it will probably come soon enough that at least our grand kids will see it.    In such a world, there certainly won't be any need to have full employment.   Probably less than 5-10% of the population would really have to work to keep the systems running. What will the rest of the world do?      And what will it mean to be rich in such an economy?    If I can have virtually any commodity goods that I desire, what would I want?   

Post scarcity economy
My more observant readers will have been shaking there head and calling me an idiot already.   I have simply described a "post scarcity economy".  People have been thinking about this for longer than I have been alive.   "Post scarcity economy"  refers to an economic state where there are more than enough resources to satisfy demand and everyone should be able to get what they need.   "Need" is highly subjective and the point where we actually enter a post scarcity economic state is subjective.    Are we satisfying all "needs" when everyone has enough to eat, or when everyone has a McMansion and a 70" plasma TV?  

The point
My point in this long winded blog post is to show that we are already on the path towards a post scarcity economy, and any solutions to our economic inequity need to take this into account.  In the United States there is already a post scarcity attitude that includes entitlement thinking and in many a lack of motivation to go out and fight for resources.  If we look at the whole world and especially the current competition for natural resources, it is pretty clear that we still have significant global scarcity, but the US is living in a bubble.     When we truly enter into a post scarcity economy there wont be a need for everyone to do traditional work for basic resources.   What will people do then?    I don't really know, but we do need to figure it out. 

The out of work problem
If we can't find work for displaced workers what will they do?    History tells us that large numbers of dissatisfied people with a lot of time on their hands tends to create unrest.   Everyone needs something to do day to day and to give them a sense of accomplishment, without that we will likely see gang activity rise, along with vandalism and crime.   

The news isn't all bad
I feel like I've been a bit negative in this whole article.  I would like to point out some benefits of a post scarcity economy.   We will have more time as a society to participate in art and culture, we could even see more of each other and socialize if we can stop looking at our phones long enough.   We could explore the solar system and beyond.  We could take our level of understanding of science, physics and biology to crazy new levels.  Getting out of survival mode and thinking about other areas can be incredibly enlightening, we might even figure out how to all get along...

Further reading
If you like science fiction and would like to read stories set in a post scarcity economic setting, or if you just want to read some fun books, I recommend the ones below.    Many of these books show a positive future, some of them show a dystopian view where technology has reached a post scarcity level, but society was unable to make the cultural leap to provide a good life for everyone.
You can see more book reviews on my GoodReads Profile.

Wednesday, June 19, 2013

Another housing bubble in the Bay Area

I moved to the Bay Area in 1994.
Source: Zillow.com

I bought my first house in 1996.

I sold that house in 2000 for more than twice what I paid for it, and moved up to Truckee, CA.

I moved back to the Bay Area in 2006 during the last housing bubble and felt that the market was overvalued, so I decided to rent.    I found a high end rental house and have been renting for the last 7 years.  I have saved roughly $600K in housing costs during that time vs owning the same house.

Record growth this year
The market this year is red hot.    Year over year home prices are up about 31%.    Across the street from me a developer built a 4,000 sqft house on a 8,000 sqft lot and sold it for $4 million.    The most expensive home on the street previously had been only $2.5 million.   The owner of the house I have been leasing noticed and has wisely decided to cash in on the current market, so now I need to find a new place to live.   I have been making the tour of houses in Menlo Park for the last two months and I'm exasperated.  Very average houses are going for well above asking price and land values are crazy.    Complete tear downs are selling for close to $2 million.  How does this make sense?    I created a spread sheet to try and analyze the market and see if it makes any sense.

According to Zillow the average house in Menlo Park is currently $1.46 million dollars, that is up from $1.04 million in May of 2011.   Keep in mind that these are averages and include condos and distressed properties and houses literally right next to highway 101.    The real truth is that you can't buy a "normal" house for less than $1.9 million and that "normal" house will be about 1,600 sqft.   The average cost per square foot is at or above $1000 sq/ft.   Any house that you might consider "fancy" is 3 million or more.

The recommended guidelines for housing costs are about 28% of after tax income.   In California, people are used to spending more income on housing so doubling the figure is about 55%.   I built a spread sheet that calculates the full cost of owning a home and will calculate the income required for 28%, 55% and 75% of income going towards housing.   The spreadsheet takes into account the mortgage interest tax deduction and the property tax deductions.

If I run the numbers for $1.46 million and $1.9 million here are incomes necessary to have 28%, 55% and 75% of income going to housing

Necessary income for $1.4 Million house
$688,198.86$322,812.16 $221,766.95

Necessary income for $1.9 Million house
$885,066.96$421,091.92 $292,782.77

Now lets look at the average income in Menlo Park.    Average income can be misleading since it also includes people who are not home owners.   The published average household income for Menlo Park is about $110,000.    If I dig a little deeper and take just the most affluent area of Menlo Park we can find that the household income for West Menlo is about $220,000.  (The data source is city-data.com)

So we can see that at $1.4 million the average household would be spending 75% of income on housing.   The real reality of $1.9 million would find that average household would have to spend 94.5% of their income on housing.

I think that it is pretty clear from the data that the average person cannot afford the average house right now.    But what does that mean?

What is causing the spike in prices
If the average person living in an area could not afford to re-buy the house that they are living in, what does that mean?   
  • The neighborhood could have a recent influx of new buyers that are more wealthy than the existing base.  (This is usually called gentrification)
  • Supply is constrained and this is temporarily pushing up prices.
  • Interest rates are about to rise and people are buying before they go up.
  • The Facebook effect?
  • The tulip craze is back.
Is it really Gentrification?
Gentrification is a possibility, but is it really happening?  How many people really make $885K a year?     Let's me more reasonable, lets allow for 55% of income for affordability, so the real question is how many people really make over $420K.    According to the IRS in 2010, 800,000 people in the US make $500K or more.   How many of them live in California?   I can't find a direct source for this, but we can estimate it.    According to the IRS in 2009 about 20% of Millionaires live in California.   If I generously estimate that 75% of them live in the Bay Area and not LA then we have about 15% of all the Millionaires living in the Bay Area.    That's 120,000 people (very approximately) in the Bay Area with household income above $500,000 per year.     So I guess this is  a possibility.  

Similar Neighborhoods in the penninsula:  (number of households)
  • Menlo Park: (12,000)
  • Palo Alto:   (25,000)
  • Atherton:   (2,500)
  • Los Altos:  (10,000)
  • Hillsborough: (3,600)
  • Los Gatos:  (12,000)
  • Woodside:   (2,000)
  • Total: (67,100)  peninsula only
Constrained Supply
This does seem to be a serious problem.   There just are not enough houses for all the people moving here.    Additionally, Prop 13 encourages people to stay in their homes and never move, so there is very little turn over in the market.   High prices combined with taxes hurt turnover as well.    If you are living in a house that is suddenly worth 2 million dollars it is very hard to sell because taxes will take half the proceeds.   Then you need someplace else to live and everything costs more than a million dollars.

Basic economics says that prices will go up if there is more demand than supply.    But how high and why now?    Supply has been a problem for the last 20 years.   Why is the market up 30% in one year?

Interest rates on the rise
I think the recent spike may be related to interest rates.   As interest rates climb the monthly cost of a home goes up substantially.    I did all my affordability calculations at 4% interest.   Here is a table of annual cost for a 1.9 million dollar house at different interest rates: 

$146,000$160,000$173,000 $188,000

That equates to more than a thousand dollars a month in difference for each percentage point.    At 7% you would spend $42,000 more per year on the same house than at 4%.    If interest rates return to normal levels of 7% over the next few years, it should cool the market considerably.

There is more than 75 Billion dollars worth of new value in the Menlo Park area.   The vast majority of that is owned by a small number of people, but there are at least a few hundred new Millionaires in the last year.

Tulip craze?
Are we in another temporary bubble?    I really don't know, but I hope so.    Nobody benefits from sky high real estate prices, except the real estate industry itself.    Surrounding neighborhoods are also experiencing record setting prices.    The Zillow Palo Alto index is over $1.7 Million and even towns like Sunnyvale and Mountain View, which are traditionally far less expensive, are approaching $1 million.     Local companies will not be able to attract talent to the area if people can't find an affordable place to live.   Keep in mind that the national average for a house is $158,300.    I would never expect the Bay Area market to be average, but I would expect it to be closer to 5x the norm rather than 10x.

What am I going to do?
I love living in the Bay Area and I love Menlo Park most of all.    The combination of an amazing climate, proximity to fantastic natural wonders like Tahoe, Marin and Napa and last but certainly not least the amazing people who live here.     If I could convince myself that the market won't fall again in the near future I would be happy to buy an overpriced home here.   Unfortunately I'm convinced of the opposite.   I think the market will correct downward at least 25% in the future, but I don't know how long it will take.

Some people called me crazy back in 2006 when I predicted a correction.   They said that real estate in California never goes down and even bet me that it wouldn't.  I won money on those bets, but I was polite enough not to collect since the folks who said it lost a lot in the down turn.    I was complacent in 2009-2010 and didn't buy then, so now I'm in a difficult spot, my mistake.

Barring a happy accident falling into my lap with an under priced gem, I will be renting again.    The rental market is just recently starting to correct closer to ownership prices, but it is still a much better value than owning, it also carries no risk of a massive loss if the market corrects back to normal levels again.

Here is my spreadsheet that has the calculations

Please leave a comment with your favorite theory and prediction for housing prices in the future.

Tuesday, June 18, 2013

Why great software artists steal rather than borrow

"Good artists borrow, great artists steal", Pablo Piccasso

It's not a perfect analogy for software, but it sure gets your attention.   A different way to say this is "Don't borrow the ideas of other products, steal the building blocks themselves instead of rebuilding them."

Software teams are able to get new products and updates to market much faster than they were 10 and 20 years ago.   Much of the productivity gains have come from "stolen" building blocks that have been made open source or integrated into the development stacks that we use.

When I helped build Netscape Navigator nearly 20 years ago we had to start at a very low level and build everything up.    The networking libraries started with TCP sockets and even then there was not a consistent API.   The UI was built up from fonts, pixels, primitive menu API's and drawing commands.    When we wanted to display a GIF or JPEG we wrote all the code ourselves.   Everything was built using plain old "C" and we had to build all of our support libraries from scratch.    Simple list structures, hashing functions, time functions, threads, better memory allocation to deal with 16 bit machines.  We eventually built a cross platform support infrastructure called the Netscape Portable Runtime (NSPR) that encapsulated all the support code into one library and let anyone build tools on top of it.   We also gave it away, so anyone could use it.     Thousands of people over the last 20 years have been building similar libraries and making them available through open source for others to build upon.

Today when we build an application we can code the UI in HTML and let the browser figure out how to render it.   We can build the business logic in a high level scripting language that handles threading, memory management and a whole host of other complex low level problems for us.   We have access to thousands of open source libraries that can do incredibly complex tasks.    On top of all that we can even host our applications in the cloud and let someone else build, maintain and monitor all our hardware for us.

New code is still hard
UI's are easier, infrastructure is easier, memory management is easier, but coding is still pretty much the same.     We have lots of new buzzwords for software development these days:   Scrum, Agile, Extreme, Waterfall.   It all sounds very exciting, but I can tell you that team programming is still pretty much the same.    I have been doing versions of Agile programming for more than 20 years and we have now codified the methodology significantly, but the nature of the game has not changed very much.    New code is still hard.   It takes longer that we estimate and getting it bug free is very hard.   Keeping it bug free while extending it is even harder.

Don't build anything you don't have to
Every single thing you build will take about 10x the time in maintenance than the time it takes to build.   The things we build tend not to be as fully featured or as well thought out as a successful open source library and likely won't have half the features of the components you could have "stolen".   

So when do you decide to build something?
Deciding when to build something new versus using something that exists is really tricky.   One of the hardest parts is knowing what is out there.    There are literally tens of thousands of open source libraries that we could grab that do virtually anything.    If you want to make smart decisions about when to "steal" a component you need to educate yourself and get to know what is available.     It may seem like a lot of time spent researching and reading, but imagine how much time you are going to save by not having to build a major component for your development project.    And how much time you will have later on by not have to fix the bugs and add features because the component you found is already feature complete and stable.    Get on the internet and ask people if searching doesn't yield results.   

Spend all your time wisely
In a start-up, and in any company, but especially a start-up, you should spend 100% of your time building unique value.    Time spent reinventing the wheel does nothing to generate unique value.    Investors won't care that you built the coolest new version of crontab to control your back end processes, they care how many users you have and how much money you can make.  

Use your components wisely
Don't try to use components in ways they were not designed to be used.   First, you will likely find bugs that no one else has encountered because they have never used that component in that particular way.   Second, it probably won't scale and perform the way you expect.   

Don't try to second guess the design unless you are an expert
Understand and follow the best practices for each component.   You might not understand all the reasons for the best practices, but they are there.    If you want to use the Zend PHP framework you are going to have to use a model-view-controller paradigm.   You may think that it's too complicated for the simple website that you are trying to get done right now and you don't want to spend the time to learn how Zend's M-V-C model works, but you need to do it.    There are really good reasons why you should do these things, but they really don't manifest themselves until you start maintaining the code or decide to change the UI later in the product life cycle.   Or you might decide that SQL is way too complicated and design an intermediate language to sit on top of it.   If you think that's a good idea and your not a bona fide SQL and relational DB expert you are going to get in trouble.     SQL got complicated for a reason, trying to second guess 30+ years of iteration is very dangerous.

Don't use components that suck
I have a whole separate blog article on this one.  "For software development tools it's the ecosystem that matters most".   Make sure all the components and tools you use are bulletproof.    Time spent fixing someone else's code takes away from your product.

Give back
On a final note, I would like to encourage you to give back when you can.   All of the code that we "steal" comes from somewhere and wouldn't be there without the hard work and generosity of those who put it there.   Recognize that and give back to the open source community by contributing new libraries and enhancements and bug fixes.    I have open sourced almost all the code I have written and it has been a highly rewarding experience.

Wednesday, June 12, 2013

For software development tools it's the ecosystem that matters most

In this article I will be exploring the different types of software development tools and how to choose the right ones for your next start-up or project.

In my career I have had the pleasure of working with several successful software teams on a variety of projects.  I have also had the opportunity to mentor or review more than a hundred software development teams. I have written programs for Windows, Mac and Unix, Android and a bunch of Web delivered apps. At Netscape I was part of a small cross platform development team that wrote Netscape Navigator which at that time was the most used application in the world.

In this article I would like to concentrate on platforms and tools. Platforms and tools are one of the least understood or appreciated since they technically focused and most engineers don't get the opportunity to choose their tools.  Whenever a new software project is created we make important choices about what platform and tools we are going to build with.  I'm defining the platform to be the hardware and operating system and the tools to be everything else above the operating system.

Platform choices used to be pretty easy:  Windows, Mac or Unix. When the world was 90% Windows, most software developers knew right away that they would be using Windows as their platform.  Today we have many choices. Win, Mac, Linux, Android, iOS, Blackberry, Windows Mobile or HTML.  Platform choices are usually driven by business reasons, but there are still technical choices to be made.  For instance should you have a native windows or Mac app?  Web applications are preferable for most since they solve the distribution and update problem, but they are not able to integrate seamlessly with the computer and can't handle long running background tasks.   Music players are a great example of this.    Lots of music companies have cloud players that run in the browser, but most choose to do a native app to support offline play and to have a fancy native UI.  Mobile app companies also have an important choice about what platforms to support. Many mobile start-ups I see are just concentrating on their mobile apps, and not creating a web based experience. Unless they are a gaming company this seems extremely short sighted.  Most of us live and work in a heterogeneous environment, so a web based experience allows us to work with a service no matter where we are. GoodReads and Google Maps are good examples of this.

I find that tools are the most difficult choices that we make at the beginning of a project. There are many different classes of development tools. These are just a few:
  • Software language:  C++, Java, PHP, Ruby, ...
  • Framework: LAMP, Zend, Rails, Tomcat, ...
  • Compilers & IDEs: G++, MSVC, Eclipse, XCode, ...
  • Debuggers: GDB
  • Databases: Oracle, MySQL, Postgres, Mongo, Cassandra
  • Bug tracking: Bugzilla, Jira, FogBugz, ...
  • Revision Control: SVN, Git, CVS, Perforce, ...
  • Integration: Jenkins, Bamboo, ...
  • Unit Testing: CppUnit, Google Test, JUnit
  • Static Analysis: Lint, Klocwork, Coverity, ...
  • Hosting: Equinix, Amazon, RackSpace, Google, ...

Many software tools are chosen on the basis of what was used on the last project.    Familiarity is an important consideration, but should not be the only one.    If we simply go with what we know we are missing opportunities to choose a tool that has advanced or is simply better suited to the current project.  Here are some important criteria to consider:
  • How suited is the tool to what I am building?
  • Will the tool handle the scalability that I need? 
  • Maturity
  • Supporting tools
  • Community

How suited is the tool to what I am building?
To a hammer the whole world looks like a nail. Perhaps you really need a wrench or a screwdriver?   If you are building a financial application, don't try to use a fancy new noSQL database.   Use a good old fashioned conservative SQL database.  If the type of app you are writing has a large portion of users using one tools set, you should probably choose that.   Why try and build an Android app with MSVC when everyone else does it with Eclipse? 

If you are building a web application for a small group, performance probably doesn't matter.   If you are trying to be the next Facebook, you might want to think about how the tools you are using will scale up to millions of transactions and a big design team.  Facebook is experiencing this problem in spades with their choice of PHP as a language.  Their improvements to PHP are very impressive, but the effort could have been avoided by choosing a different language.  Scalability also applies to team programming.    Dynamically typed languages, such as PHP, Ruby and Python can be challenging for very large software projects since the functions are not explicitly typed.  This makes it hard to detect software problems at link time and requires engineers to document their code in ways that are not enforced by a compiler.

Tool maturity is too often overlooked.  I think this one should be easy, but I see start-ups making the wrong choices over and over again.  If a tool isn't 99.999% stable, you probably should not use it.  All of your time should spent solving technical and business challenges.  If a start-up needs to spend time debugging their tools they are wasting valuable time and resources solving a problem that they didn't need to solve.    How should you evaluate maturity?   Past personal experience or a referral by someone you know and trust is best.   Otherwise, look to see who else is using it.    If you can't find a large list of other companies that are using the tool in a production environment, stay away.    Even the most promising tool can turn into a nightmare if you make it part of your product and it starts to crash or otherwise misbehave.  Building tools is hard, not just because they are hard to build, but because they are very hard to perfect. You need tools that work every time, not just most of the time.

Supporting tools
This is probably the most important tool trait. Every tool class listed above comes with an ecosystem of tools that support it.  With a database you need backup tools, profiling tools, monitoring tools, API libraries and documentation.  New tools often look promising until you look under the covers and find out that it is lacking basic support infrastructure.  Do you really want to find out after you go into production that your tool needs to be taken offline just to back it up?

Profiling and debugging tools are critically important and are often the last thing on a product road map.  If you have ever experienced a critical performance bottleneck and have been without a real profiling tool you will understand how important a profiler is.  How about finding a thread deadlock with a debugging tool that doesn't properly deal with threads?  Memory corruption detection without a dedicated memory debugger?  These are support tools that usually take many years if not decades to get right in a major tool class, but they are absolutely necessary.   

Last but not least is the community of people that can support you for questions and problems.  The absence of a strong community should be a huge red flag. Poke around and ask questions, make sure that there is still active development of the tool that you are choosing and lots of active users.  Look for example code and places where people congregate to ask and answer questions.   You will save a lot of time if you can draw upon the community to figure out problems for you.  Open source libraries built for your tool can also make your job a lot easier.  Plugins to support your other tool choices and layer in great product features.

Looking at the whole stack
Now that you have a framework for evaluating individual tools, you now need to go up a level and look at your entire tool stack.  Almost all of the development tools you use will interact with each other so they need to be compatible.  Want to use a Java based messaging framework but you use C++ everywhere else?   Think again.  Love that C++ static analysis tool, but you use Python?  Sorry no dice.  Your choices on one tool effect every other tool you will use.   Make sure that you evaluate the whole stack before you commit to any one tool.  Otherwise your one choice will likely dictate and limit all your other tool choices.    

Current Experience Levels
Last but not least please be realistic about your team's experience relative to the tools you are choosing.  It might seem like a great idea to use Ruby on Rails for your next major web project, but if there is nobody on your team that has any experience with Ruby you are probably going to run into trouble.  If you already know PHP the benefits to moving to Ruby are probably overshadowed by all the mistakes that you will make learning a whole new framework.  Remember that the end product is more important than the language used to express it and the user isn't going to care if you are using the latest and coolest technology to pull it off.  Conversely, using a technology that very few people know, even if your team is very proficient with it, will limit your ability to hire new people in the future.  

Apply this same methodology to support libraries as well
All of the points above also apply to support libraries that you will use in your product.   Here are a few examples.
  • Logging
  • Messaging frameworks
  • General Utility libraries such as boost for C++
  • Caching and proxy libraries and servers
  • XML & JSON libraries

Look at all your tools carefully before you start using them.   Make sure that they are all of the highest quality and maturity and that they come with the supporting infrastructure to handle the whole development process:    Coding - Testing - Debugging - Profiling - Monitoring
Doing this will save you and your team countless hours of frustration relative to less mature tools.

Monday, June 10, 2013

An HTML5 fancy user interface project

Visualizing disk space with a sunburst graph
Sunburt graph
I recently finished a new project that uses HTML5 to display and graph file system data and I thought I would write a blog post about it.
HTML has come a long way from the early days when I was working on Lynx. The latest and greatest features are pushing graphics and user interfaces that are approaching native code levels of complexity and performance. I'm particularly fascinated by these features because they represent the realization of a vision that we had in the early days of Netscape where HTML could take over as a complete application programing interface.

WWW applications?
Using the Web just like a computer program seems pretty obvious these days, we use it that way on many if not most web sites, but in the beginning it was really just a document publishing platform. Hypertext, the basis for the web, was designed as a way to make documents more powerful by linking them together with references that could be accessed incredibly easily.  The idea of putting powerful application logic behind a hypertext interface was pretty far fetched in the early '90s.  I happened to have a need to do just that due to a need at my university. I came up with some ideas and put forth an idea to the other developers I knew that were working on the web. This idea soon sparked the addition of form elements to the web. Later while at Netscape, we took the next step by adding a scripting language, Javascript, to add even more client side capabilities and responsiveness.

Full GUI support
One of the visions we had at Netscape was to create a platform that had a fully complete and functional set of graphical elements so that other developers could create any application imaginable. The foundational elements are reasonably well known and are included in all modern operating systems and major graphics libraries.  The difficulty in delivering GUI primitives to the Web was figuring out a way to efficiently add them to HTML and to make them massively cross platform.    When I say "massively cross platform" I don't mean just Windows and Mac. We were thinking about operating systems, screen sizes, resolution, and a whole bunch of other things.  The web was designed from the ground up to try to be device neutral and that makes the implementation very hard.  CSS is one technology that supports multiple displays by providing different layout and styling for different displays as required.  Netscape as a company ran out of life before it was able to complete its roadmap for a full complement of GUI primitives, but it did make a good effort.  A technology called XUL (pronounced Zool) was introduced that allowed a large variety of GUI to be expressed, unfortunately, XUL didn't move forward fast enough and was essentially dead on arrival.     Simultaneously Microsoft smothered the market with Internet Explorer and tried its best to keep the Web from moving forward with HTML.
Visualizing disk space with a tree graph
Tree graph

Enter HTML5
Fast forward several years from the Netscape era and the world has changed substantially.    Firefox, Chrome and a variety of Webkit based browsers have wrestled away the strangle hold that IE6 held on the market and began to make substantial improvements to the abilities of HTML.  HTML5 was soon born and with it a whole bunch of new graphical technologies that enable many new types of GUI applications.    SVG, advanced CSS, Canvas and Javascript now combine to create an ability to create very rich interfaces that are entirely driven within the Web browser.

My experiment  
A while back I came across a group that was creating a set of tools that harnessed the new power of HTML5.  The project is called D3 for Data Driven Documents.  I saw what D3 could do graphically and wanted to try it out and do something personally useful with the technology.   Since my current company Zetta.net is in the cloud storage space I decided to write a tool to visualize the size of the data on a computer hard drive.  One of the difficult issues with a tool to analyze disk space is that it needs access to all of the data on a drive and that data is sensitive and private.  I needed to figure out a way to scan the data and not violate a user's privacy. Standard HTML applications run in the cloud and couldn't do something like this without sending data to a server, but the new world of HTML5 can make everything happen locally in the browser.

Some complications
At this point in my project I had an idea of what I wanted to do, and a fancy new graphical library for graphing, but I still needed a way to collect the disk space data to be graphed.  Javascript has chosen to stay away from local file access to simplify the security model, but Java long ago added the ability to ask the user for local disk access.  I ended up writing a Java beans integrated module that spawns a Java application to scan the local disk and return the data to Javascript running in the browser for analysis.  The beauty of the Java/Javascript integration is that none of the data ever leaves the local computer that it is running on. The downside is that Java is often broken on user machines, or is disabled due to security concerns.  I hope at some point Javascript figures out a workable security model to allow some forms of local disk access.
Visualizing disk space with a tree map graph
Tree map graph

The finished project
Please check out the finished project if you are interested. I have made a video demo and a walk through to explain the use and features.   There is also sample data to play around with that skips the scanning process.  The code has been published as open source with a BSD license.    This project just might come in handy for you the next time you run out of disk space and need to know what is using it all up.


Wednesday, June 5, 2013

A phone for every feature. Where is Apple?

6/12/2013: Update:  The Galaxy Zoom was announced today.   This is a pretty great looking device.    If you are serious about photography and have been carrying around a Canon s100 plus a phone like I have, this could potentially make your pockets lighter.  (in two ways)

Today Samsung announced the Galaxy S4 active, a dust proof, water resistant, rugged phone.    Motorola is talking about the new X phone, a phone with a battery big enough to last for days and a host of new sensors to know when it's in your pocket, traveling in a car or just hanging out.    The latest generation of phones from Samsung and HTC include infrared transmitters and NFC and a screen sizes for every hand size and preference.  I'm overjoyed with the choices and features in phones these days, but in all of this I have to wonder where is Apple?  The former leader in phone innovation which introduced and dominated the smart phone market seems to have quietly slipped into an underdog role.   

A bet on the future
A few years back I had a thoughtful debate with a VC friend of mine about the future of the phone market.  I had just gotten a shiny new HTC Android phone and was a member or a small minority.    I was looking at the fairly unpolished features of Android 2.3 and seeing all the untapped potential.   My friend was basking in the glow of the polished iPhone experience and confident that Apple would continue its dominance in the smart phone marketplace.  Being an open source advocate certainly influenced my thinking, but I was confident that the open model would eventually produce a phone that would outclass the iPhone.    I was so confident in my new Android phone I bet one whole dollar that Android would eventually rule the market.  (wow!)
Android turns the tables
New Android phones today ship with a version of Android (4.2) that is significantly more polished than a few years ago.   Android is now a legitimately beautiful and elegant solution that still offers the flexibility and power to customize your phone endlessly to your own whims.
Android phones currently outsell the iPhone 3 to 1 in total units, but the iPhone is still a dominant player in the premium market.  Here in Silicon Valley most of the affluent people I know are still iPhone users and Apple still has a fantastic brand and an ability to move the entire market, but how long will that last?   If Apple continues to sell only one form factor and one premium phone can it really compete in a marketplace full of specialized choices?   Should I have to wrap my phone in a big case if I want to make it rugged?  Should I have to add an external battery case if I want the battery to last?  I want choices and I want my phone to fit my lifestyle.

What could Apple do?
I would like to see Apple learn from its previous mistakes and open up its ecosystem with other hardware vendors.    I want iOS to be set free.   I believe that if Apple had started licensing iOS several years ago and satisfied the demands of hardware manufacturers it would never have lost market share to Android and would control nearly the entire mobile market.    As a consumer I'm quite happy that there isn't one dominant closed source vendor, but from Apple's perspective iOS is really missing the market.    Waterproof rugged iPhones,  iPhones with 6 inch screens and a whole host of other features would be fantastic, and I would expect that at the top of the food chain Apple would still be selling the highest quality, highest margin iPhone.

Now that I've said it, forget it.
Apple will never do this.   They will argue that it would make it impossible to have a seamless experience, that they tried it before with Mac clones and it failed then, and that they are a hardware company.    Too bad, opportunity lost.    My prediction is that Apple's mobile market a few years from now will look a lot like their desktop market.   10-20% market share on the high end and no real influence in the true direction of the market.   They will still command high prices with reasonably high margins, but they will not have a dominate position in anything.

Is Apple losing market share a good thing?
Yes!   A market controlled by Apple or any other closed ecosystem vendor would be very scary to me.   Apple has a history of telling it's users what they want and ignoring market data.    Apple also is trying to extract high margins from its products and benefits by keeping prices high.    Fortunately for us the Android alternative is here and has fantastic fundamentals.    Android is, at its core, a true open source and truly open technology.   Android is built upon Linux, and most of the code is released under the Apache license.  That means that we can take the code and use it even if Google doesn't want us to.    There are parts of Android that are closed and proprietary, but the meaty core is open to us and essentially free forever.   So lets say Google goes all evil on us in the future?   Will they be able to pry my phone from my cold dead hands?   No, we the consumers, and any other company that wants to, can take the core of Android and release something even better.    

OK I'm biased, I admit it
Yes, I admit it, I'm shamelessly writing a blog article about a subject that people just love to argue about.   I'm also a huge open source bigot so I'm naturally inclined to like Android.     I really just wanted to write about how much I love the progress that Android has made and the fact that for the first time in history an open source operating system has dominate market share.   It really is about time!

Flame on!
Feel free to start a debate in the comment section, but please try and put forth reasonable arguments.   Nobody benefits from name calling and bickering.    I'm sure lots of people will disagree with my predictions, if you have your own, call them out and tell us why you think they are right.

Thursday, May 23, 2013

Value based home music recording

Value based credentials
Everyone who knows me knows that I'm obsessed with getting a good value. I probably look for good deals a little too much. My wife has said to me on multiple occasions that she would really appreciate if I would stop telling everyone how much I like to get good deals. To put it bluntly, I'm a bit cheap sometimes, although I do prefer "value based" or possessing "Yankee ingenuity". The whole idea of being cheap isn't consistent with the rest of my life, since I have been known to make a fair bit of money and have spent quite a bit of it as well. The long and short of things is that I like to get the best value I can from the money I spend and I spend a fair amount of time researching things to find the right items to buy. I should probably write a separate blog just on my neurotic need to find good value, but let's just establish for now my credentials for finding good deals.  

The challenge
About 9 months ago I started the process of recording a set of songs. I'm not a particularly good musician and the other guys in our casual band are definitely not professionals either, but having set out to do something I always try to do as good a job as I can with the time and resources available.

The band
Our band is not a traditional band. We write and perform parodies for our school auction. The idea started with my good friend Jeremy and fellow school parent David who decided, off the cuff, to do a few songs during the school live auction just to be a bit different and entertaining. I joined the band the following year and it grew from there. Each fall we get together for 3 or 4 nights and write some new lyrics to classic kitschy tunes and figure out how to hack our way through them. Our lack of great talent is part of the charm of the whole affair. (Or so we like to think) I mostly think that Jeremy's jovial personality and overwhelmingly funny and positive stage presence is the only thing that makes it work. So the raw talent that we are working with is limited, but there is a fair amount of enthusiasm and positive energy that we want to capture in a musical album of the songs we have performed over the last few years.

Knowing that I need to take the reigns and figure out how to record our band, I set out to find out how the process works and what equipment I would need to pull it off.     I'm a drummer and I have always had the additional task of managing the PA and mixing equipment.    At this point, having done it for many years, I'm pretty good at setting up microphones, speakers and mixing consoles.   I have run audio boards for other fairly decent bands in the past so I know the live performance side of things well.     Recording is a different beast and requires different tools.    Here are the things that I knew I needed:
  • All the standard live equipment: Instruments, amps and microphones
  • A hardware device that takes analog input from the microphones and other instruments and translates it into separate recordings of each instrument.  Each one of those is called a "track".
  • A computer program that can take those separate tracks and mixes them together to create a master track that is the finished product.    Ideally this software would work similar to Photoshop and can be used to touch up and improve the tracks after they are recorded.

Microphones are a bit like your favorite food or wine.   There is something for everyone and everyone will have a different opinion about what quality, price and sound they prefer.    Microphones are a cheap as a few dollars and can run into the thousands for bespoke models for professional studios.    If you already have a band, you probably already have some microphones.    The singer will probably care the most about what their mic sounds like.    If you don't have any mics I recommend looking at Shure, AKG, Sennheiser and Audix.    For this project I used Audix OM-5 microphones.   They are an inexpensive durable mic made for live giging.   If you are only going to do studio recording you can probably find something less durable and better sounding.   I also used a Shure wx-20 headset mic for myself (the drummer).    There are specialty mics you can get for guitar amps, bass drums, snare drums, harmonicas and every other type of instrument.     One mic I really wish I had was something like this AMT 15G for recording acoustic guitar.    A regular mic on a boom stand can work great for guitars, but the guitarist needs to have the discipline to not move around during the recording session.    A mic attached to the guitar solves those problems.    I don't really like the sound of the mics and pickups that are inside the guitar body.

Recording drums is an art unto itself.   Ideally you would mic every drum head and each cymbal separately.   Practically speaking most people just can't get there.    If you want to record an acoustic drum set properly you will need to spend more than I did on my entire recording set up.    I got around this problem by using a Roland TD-30.   High end Roland drums are pretty amazing to play, they really reproduce the feel of an acoustic set.   In live settings they are definitely a different type of instrument and don't feel quite as edgy and real, but for a recording session, they are really hard to beat, and the benefits for recording are amazing.   Everything you play will be recorded properly and if you record using MIDI you can go back and tweak the track losslessly.   You can also tailor the drum sounds to the music you are playing.    I'm a pretty weak drummer, but the TD-30 makes me sound better than I am.  (Don't worry, I'm not planning on quitting my day job)   I didn't buy the TD-30 for recording, I bought it so that I can play in my house everyday without driving my family crazy.    If you don't have an electronic drum set to record with I recommend two things.   1)  Isolate the drummer as much as possible from the rest of the band so that the drums don't bleed into the rest of the band's mics.   2) Record with a single overhead mic to capture the entire sound in one track.    You will have to play with the height to get it to sound right and the acoustics of the space you are in will color the sound, so you might have to figure out how to soften the room by hanging blankets or carpet on the walls.

Multitrack recording hardware
Your microphones have to plug into something.    At a live gig you plug all the sources into one mixing console and have someone adjust the levels of each source to sound reasonably good.     Mixing a live gig is very subjective and results vary wildly.   The nature of the live gig is that people expect to hear a more raw sound and it is often played very loudly which gives music a very different feel.   When making a recording, listeners tend to expect a more polished sound and they tend to be more critical of the recording since they listen to it over and over again.    In order to get a more polished sound it is necessary to get a clean recording of each instrument and save each one separately so that they can be mixed together into a final form later.     Traditionally, artists recorded onto tape.   8 track tapes from the 70s were are a commercialization of technology that artists used to record their individual tracks.    Today most people digitize into a computer and store each track as a file on disk.    There are lots of different products that do this.   Some are fully integrated and look like a mixing console, but also record each track to an internal hard drive.   A lot of products use FireWire to record to any computer often times a Macintosh, since they have supported FireWire for a long time.    Recent products use USB to interface with the computer and are compatible with any computer you have.  (PC, Mac or Linux)     I wanted to find a USB product since FireWire is hard to find these days and is almost impossible to get on a laptop.    The product I chose is the M-Audio C600.    The C600 can record 6 tracks simultaneously and comes from a company that has a high regard for audio quality.   There are lots of similar products out there from Tascam, Behringer, Roland, Alesis, Mackie, Motu and others.    The products are improving and changing quickly, so as you are reading this, chances are that there is or will be better products out already.   The product class is called a "USB audio interface".   Here are the basics of what you should look for:
  • Number of simultaneous tracks that can be recorded.
    • 1 is the minimum you can buy.   You can buy multiple single track recorders and attach them all to a computer to get multiple tracks.   This can work, but can also be a source of frustration if your computer can't handle the multiple hardware sources properly.  The cheapest ones you can buy are on Ebay for $12 each. 
    • Figure out how many you tracks will need at one time.   The bigger your band the more you will need.    I figured that I would need at least 6 to mic a 4 piece band.  
    • Figure out how many you can afford.   This technology is getting cheaper all the time, but there is still a pretty significant price jump from the 2 up to 6 and again up to 8 or 12 inputs.
  • Overall quality.   There is a big difference in the mic pre-amps between the high and low end of the market.    Read the reviews carefully before you buy a bargain priced unit.
  • Phantom power.    Transducer mics require power to operate and your audio interface should supply it.   You can plug your mics into a mixer and then key off the audio inserts on the mixer, but that represents one more step for noise to creep in.
  • Rack mount or desktop.   The higher end units and especially the 8-12 input units are usually rack mount and integrate into a "professional" studio environment.    
  • Headphone and audio sends.      Your audio interface will be used to provide monitoring for your band as you are recording.    The audio sends can go to speakers or you can use the headphones as a monitor.    Headphones work best since you don't want to have extra sound sources in your recording room.
Almost any reasonable computer should work.    I used a Thinkpad X220 laptop.    You will only need about 30 gigs of free disk space for most uses.    Obviously if you choose a Firewire solution, you need to have a computer that has a Firewire interface.    If you have a USB audio interface, you can use pretty much anything.    Almost everything is USB 2.0 still.   USB 3.0 is coming but isn't really necessary for most uses.

Software is where you can really spend or really save, the type of software you need is call a DAW or "Digital Audio Workstation".   The 800 pound gorilla in the space is Protools.   Protools is what the pros really use and has pretty much every feature you could imagine.    Protools is expensive and has a really onerous DRM system that involves physical keys that you have to plug into your computer, it also retails for about $700 for just the base software.   There are "starter" versions of Protools and I received one with my M-Audio C600, but it turns out that the way they restrict the usage is to limit the number of simultaneous tracks you can record to two.    With only two simultaneous tracks you really can't record a band, you can only record yourself.   I'm an open source software guy and have given away nearly everything I have ever written, so I looked around to see what I could use.    I tried to use a product called Audacity.   It seemed to be the best of the open source tools, but it had a real problem interfacing with my hardware.    Most of the audio interfaces work best with a Windows driver interface called ASIO, and I couldn't make ASIO work with Audacity.    There are many DAW's out there, but I really wanted something that had a bit more polish to it and supported ASIO.   During my research I came across Reaper.   As it turns out Reaper was everything I was looking for.   It is a nicely polished product that works very reliably, supports ASIO and has very attractive pricing.    Reaper is free to try and nags you if you don't buy it.    Once you decide to buy it there are two options.   It's only $60 if you are a small time user like me, and $225 if you are a big time commercial outfit.    Most of us will fit into the $60 tier and that is a tremendous value in the DAW space.     One of the biggest benefits of Reaper, besides that fact that it works fast and reliably, is that it supports VST plugins.   VST plugins are the same things that make Protools such a powerful platform.    The DAW software is not just a recording, mixing and editing solution, it is also a platform for adding lots and lots of 3rd party effects to make your tracks sound amazing.

The recording process
Every band will do it different, but here is what I experienced.   We only had about 8 weeks from the start of the process to being completely done and we were trying to record more than 10 songs.    A real band would take longer than that just for one song in most cases.     On many of the songs we recorded the drums and rhythm guitar in one session and added the rest of the instruments as overlays later.    In some songs we tried to do all the instruments together and then went back and rerecorded some parts that we didn't like.    The complications that we ran into were many.     Sometimes one or two of our band members were not available, so we had to try and make progress.    Sometimes we would all get together and waste a bunch of time just trying to get ready.    We had problems with bleed when we tried to record multiple acoustic guitars.     Most of us didn't like our performance and wanted to go back and try it again and again.    The most important thing we did, was to keep making progress and keep making recordings.    Eventually we were able to get about 12 songs recorded in various states of completeness.   

After the band had recorded the major bits, I went back and overlayed backup vocal tracks, bongos and other percussions and started playing around with effects to try and make us sound better.

VST Plugins
After you capture your recording you will start mixing and editing.   A short while later you may  realize that you want to add effects to voices and instruments to enhance the sound.    VST plugins is where to turn for amazing effects.    Start searching the internet for VST plugins and you will find hundreds of choices for free and commercial plugins that do virtually everything.    Want to get that Autotune sound to be just like T-Pain, you can get the original commercial version and plug it right in.    Some really great free plugins are gVST and BetaBugs

Final cut
You may be listening to your songs on headphones and love the sound only to be surprised later on that it doesn't sound nearly as good in your car or on your iPod.    It turns out that one of the secrets of good audio engineers is that they know how to mix a song so that it sounds pretty good on all audio systems, even though it won't sound perfect on any of them.    One of the secrets is that they compress the music substantially.    Compression removes dynamic range from a song, but it also makes it sound louder overall in general.   It raises the volume of the quite parts and lowers the volume of the loud parts.   Compression is very controversial in the music world, but I will tell you that if you don't do it, people will probably tell you that your song doesn't sound right, since they will probably be listening to your song with poor quality speakers.    If your audience is all audiophiles or people with high quality headphones then leave out the compression.

Total cost and equipment list

Here is the list of equipment and approximate price for them:
  • $0  A laptop or desktop computer (assumed that you already owe this) 
  • $250  M-Audio C600
  • $360  3 -Audix OM-5 microphone (3 x $120)
  • $80    Shure WX20 headset microphone
  • $200  Mic stands, cables, beer holders, etc...
  • $60    Reaper DAW software
Total cost, just under $1000 USD.   The experience of recording and producing music, priceless.

The result
I know it's not terribly good, but I'm proud of the result and we had a great time making the music.   I hope these tracks are received in the spirit in which they were recorded.   I'll just post a few.

All of these songs were written for our nursery school, a COOP school in Menlo Park called the Menlo Atherton Cooperative Nursery School.   

Stop. Get Right Down and work it out
    This is a song about two toddlers battling it out on the nursery school playground.   It was inspired by a parents meeting in which we all learned the best way to try to resolve conflicts between two  children on the playground.

Can't you see what that COOP means to me
     This is the last song we wrote and is a look back at how the school had effected our lives and how special the time was.     Listening to it now brings back fond memories of all the friends I met and the great experiences helping to teach the kids in the school.