How to pick an ETF? Hint: Don't focus on cost

As individual investors buy into ETFs, there's one very compelling aspect: a low cost relative to many traditional mutual funds. But for ETF experts, cost is rarely ever the deciding factor in making the decision to buy an ETF, especially now with so many ETFs in most major asset classes.

Cost is a factor in selecting the "right" ETF—fund costs eat into potential returns—but the best returns will come from ETFs offering an advantage across a number of variables, including the uniqueness of fund strategy relative to others in the same asset class, liquidity and trading costs.

Pgiam | E+ | Getty Images

"If you sorted the best and the worst performers over a one-, three-, and five-year period, the best is not always the cheapest, and worst is not always the most expensive," said Robert Holderith, president, Emerging Global Advisors, an ETF provider. "Some investors are purely price driven, which I think is a mistake," he said.

(Read more: Fight the Fed with these bond ETFs)

Take the rise of smart-beta strategies in the ETF world—or in layman's terms, index strategies that weight securities in a different way from standard market cap–weighted indexes like the S&P 500. "A better beta fund that charges more is better than a traditional one that charges less, because an intelligent beta provides an edge that is greater than the difference in cost," Holderith said.

Understanding total cost
David Mazza, head of research for State Street Global Advisors' SPDR ETFs—which offers some of the largest, lowest-cost ETFs in the market—said that while an ETF may look cheaper on paper or at first glance, it could be a lot more expensive when it comes time to trade it.

ETFs trade on an exchange, and investors must pay a commission if not using one of the newer commission-free discount brokerage trading accounts, as well as a bid-ask spread. (The bid is the price that someone is willing to pay for a security at a specific point in time; the ask is the price at which someone is willing to sell. The difference between the two prices is called the bid-ask spread.) The lighter the volume in a particular ETF, the higher the potential for a most costly bid-ask spread.

"If you hone in on one particular measure, such as expense ratio, when doing due diligence and homework, you may miss out on the opportunity to have a less expensive total-cost-of-ownership experience," Mazza said.

Another component ETF investors should be aware of is tracking error, which is often overlooked but should not be, since most ETFs are passively managed and seeking to track the performance of a benchmark. "A fund at first glance may be very inexpensive, but it may be quite small, so the portfolio managers can not buy all the securities the index represents, and that could introduce tracking error," Mazza said.

(Read more: Fitting a star fund manager inside an ETF)

Dave Nadig, chief investment officer at ETF.com, provided an example of how even index ETFs covering the same sector can diverge to a significant degree. In 2013 the best-performing U.S. financials ETF was the RevenueShares Financials ETF (RWW), which returned 42.04 percent, while the worst performer was the Powershares KBW High Dividend Yield Financials ETF (KBWD), with a return of 18.35 percent. That's a whopping 23.69 percent difference.

"No one can tell you exactly which portfolio will outperform, but you can't assume you are making the right choice by just looking for the one with the cheapest expense ratio," Nadig said. RevenueShares ETFs are weighted based on top-line revenue. The Powershares KBW High Dividend Yield Financials ETF is weighted based on dividend yield.

Buying the biggest or most liquid ETF is no panacea either. Nadig used an example from one of the biggest ETF providers, the SPDRs, to make his point. The largest financial sector ETF, with roughly $17 billion in assets, is the Financial Select Sector SPDR Fund (XLF). It saw $4 billion in flows last year alone, but it has consistently underperformed the $1.3 billion iShares Dow Jones U.S. Financials ETF (IYF), which has less assets and lower trading volume, according to Morningstar data.

(Read more: It's a good time to be an ETF bad boy)

For the three years ended on Dec. 31, the cumulative return for IYF was 46.5 percent, while the XLF came in at 44 percent. "Of course, time periods matter here, and you can find windows where the opposite is true," Nadig said. But bigger is not always better. "I don't believe the biggest funds always provide the best performance. It is all about strategy and not about fund size," said Emerging Global Advisors' Holderith.

The SPDR financial ETF tracks the S&P Financial Select Sector Index, while the iShares financials ETF uses the Dow Jones U.S. Financials as its benchmark.

Nadig said he wouldn't suggest that IYF will always outperform—recently it hasn't—but the difference in their portfolios is primarily in diversity: IYF has 266 stocks compared to XLF's 80, which pull just from the S&P 500. The iShares financials ETF has a significantly lower average market cap as a result: $84 billion vs. $103 billion. IYF also skews a bit "growthier", Nadig said, with a higher P/E and has some slight sector skews: it underweights banks and insurers a bit in favor of REITs. But IYF is the ETF.com pick largely because of its significantly broader portfolio.

There are big notable among the top 10 holdings of the indexes—the Dow Jones index includes Visa and MasterCard among its top 10 holdings, while the S&P Financial Select Sector Index does not.

Overall, Nadig recommends that investors look at three key areas when considering an ETF: efficiency, tradability and fit.

"For efficiency, the question is, Can I trust the ETF to do what it says it will do? For tradability, it's Can I get access to it in the marketplace, and can I get in and out of it—and finally, fit, meaning: Is this the best pair I can put on? ... Every decision comes down to these three factors," Nadig said.

The information is not hard to find. ETF companies make information on trading volume, bid-ask spread and index details available on their websites. "Investors can go through a checklist of what is important to the ETF due diligence process," Mazza said.

"Liquidity, tracking error composition, and the factor being weighted—market cap or dividends, for example—those are a lot more important," said Kim Arthur, CEO of Main Management. "Cost will be the driving determinant only if we have covered all of those other pieces," Arthur said.

By Leslie Kramer, Special to CNBC.com