Wednesday, October 8, 2014

The HyperLogLog structure: estimating the number of distinct values on "Big Data" with minimal memory

Imagine you have the following scenario: you want to count the number of "distinct" values that you have seen from a stream of data without actually storing all of the values. (For example, you want to answer a question such as "how many distinct IP addresses have made requests to my server" on the fly.) Conceptually, you want to place all of the values (in this case, IP addresses) into a set and then count the size of the set. But you want to do this efficiently on a potentially huge range of possible values (so that keeping the entire set in memory is not viable and placing the values in a database is not practical), even at the expense of the count being approximate.

For a moderate number of possible values, one potential implementation that I have written about before involves using a bloom filter (which is essentially an "approximate set").

But a more memory efficient solution comes in the form of the HyperLogLog algorithm. Effectively, this algorithm allows you to estimate the number of distinct values seen by gathering broad statistics on the hash codes seen so far. To use an easy-to-understand analogy presented in the article, if I told you that I had tossed a coin n times and got a maximum run of 10 heads in a row, you would estimate that I had tossed the coin fewer times than if I told you that I had got a maximum run of 20 heads in a row, and you could use some statistics to estimate n. The same essentially goes if I were to say, "out of all the hash codes of values seen so far, the maximum run of 0s in the lower bits was x", and I can take the average of various statistics of this type to hone the estimate.

It may not be used often for all applications, but if you are in the business of "counting events on big data", then the HyperLogLog algorithm appears to be a very worthwhile weapon to have in your arsenal. I'm sure I'll be adding it to mine.

An example of simple design at its best

I tried out the iOS game Hexagon! today. If you haven't come across it yet, I think it's an interesting example of how a simple but carefully implemented design can be extremely effective.

Thursday, September 25, 2014

Getting hit by the iOS 8 bug

If you haven't already, there will probably be more pressure to update to iOS 8 this year than there has been with similar updates in previous years. Out of the box, you may not immediately notice much of a change after updating from iOS 7 (which, by contrast, marked a major visual redesign compared to its predecessor). But iOS 8 is packed with new features for developers, which will slowly start to trickle into apps over the coming months if they haven't already. There are probably a large number of cautious laggards who would prefer to wait for things to "bed down" before upgrading. But sooner or later, you may well find that your favourite app can no longer be updated without upgrading to iOS 8. In the meantime, most users have surely noticed the flurry of updates arriving in the app store as developers scramble to fix issues that have come to light as a result of the iOS update.

One of my own apps illustrates the kind of subtle issue that developers are having to grapple with. The Utter French pronunciation guide, as the name suggests, makes fairly extensive use of audio playback. Like many, I tested the app using the iOS 8 beta, but one slightly subtle bug did pass me by.

As mentioned, the iOS 8 issue I discovered relates to audio playback. The issue is caused, not so much with the audio per se, but rather what happens after a sound has been played back. At the end of audio playback, the system notifies the app via what in Apple speak is called a delegate method (in effect, a callback to what is akin to an interface method implementation in Java). The system makes a call to the programmer's implementation of the audioPlayerDidFinishPlaying playback. method to indicate the end of playback. Crucially, it passes in a boolean parameter to indicate whether playback was "successful". Subtly, between iOS 7 and iOS 8, for reasons not yet clear, the value of this flag changed from TRUE to FALSE in the cases I observe, despite the fact that audio apparently plays back correctly. Other developers have also reported this issue. The reasons and circumstances remain unclear: perhaps iOS 8 has tightened up on some subtle features of audio files that it now deems to be "incorrect". Or perhaps it's just a bug that will be fixed in a subsequent maintenance release of iOS 8. But the knock-on effect for my app was that it took the FALSE flag is a signal not to play back the remaining items in a list of audio files queued for playback.

The workaround-- currently in review and hopefully to be released shortly- is simple in this case. But it illustrates how a very tiny change in the behaviour of an API call can have a knock-on effect. Multiply this kind of subtle change by who knows how many system functions and we start to see why developers are scrabbling to release updates and fixes for iOS 8...

Despite this, my advice to most users would still be to update iOS 8 fairly soon. The stream of daily updates in the App Store indicates that developers are getting on top of the issues, and it won't be long before a new app or update comes out with an iOS 8 only feature that you want!

Saturday, August 16, 2014

Horses for courses: Stroustrup's InfoWorld interview

In a recent interview for InfoWorld, C++ creator Bjarne Stroustrup talks about why he believes C++ is still going strong in 2014. To me, two statements of his argument stand out:

  • he attributes the popularity partly to the fact that "nothing that can handle complexity runs as fast as C++";
  • he acknowledges that "C++ is designed for fairly hardcore applications" and that it can be part of a mix of different languages (he mentions the fact that he himself uses C++ along with a scripting language such as Unix shell script).

Taken together, these points are broadly fair. What I do wonder, however, is to what extent he is characterising as intrinsic language features what arguably are more compiler features than language features. And inasmuch as C++ forces you to as a programmer to get a little "nearer the metal" than a language such as Java, to some extent it does so because it is based on C rather than because of the object orientation and other features added in C++.

It's also fair to say that not all uses of C++ historically have been for "fairly hardcore" applications and that, as at least one commentator on the site has pointed out, some of the popularity of C++ is surely attributable to the fact that, once a large-scale application is written in one system, it's difficult to find the momentum to shift to a whole new language or development system.

All in all, though, the interview does go to highlight that language wars are largely futile. Languages like Java have a strong footing in their specific domain. And C++ does in its domain. What is more important is to focus on the right tool for the job.

Sunday, August 10, 2014

What are the top 10 programming languages? (And is HTML a programming language anyway?)

This IEEE survey, based on code available in various repositories, concludes that Java is (by a short margin) the most popular language occurring in these sources. If you had expected choices such as Ruby and Objective C to be front runners, then you may be surprised to find that overall, the list of top languages has a distinctly "old school" feel to it: C is a close second to Java, followed by C++ and C#. Objective-C comes a distinctly underwhelming 16th.

We will avoid a debate here (but the article comments needless to say have not) around whether choices such as "HTML", "MATLAB" and "R" count as programming languages.

Of course, the choice of languages reflects the specifically the choice of languages occurring in code repositories rather than the interest in these languages across the industry as a whole. It's frankly improbable that there are more MATLAB programmers than iOS and Mac OS programmers and the vast majority of programmers in the universe have probably never even heard of R. What I suspect this graph is showing us to a large extent is the relative proportion of programmers in various languages who share their code in repositories versus those who keep it under their hat.

Still, for those of us who started learning to program in the 80s, it does seem to indicate that our trusty C skills are not going to be obsolete any time soon... :)

Saturday, August 9, 2014

Will new iOS device sizes open up new markets for developers?

A pertinent point is raised in this Wired article on the new iOS device sizes that are apparently coming later this year. In countries like the US and the UK, where the use of multiple devices is typical, the availability of a larger iPhone or smaller iPad is arguably not a game-changer but simply a "small part of the mix".

But in other countries where the mobile phone may be more of a primary device for users, the availability of the phablet form factor may more strongly influence users' choice of platform. For Apple to have such devices as part of its mix may then open up new markets for developers. As I highlighted in a previous post, the option in Xcode to test our apps on arbitrary screen sizes should probably not be overlooked in the coming weeks...!

Thursday, August 7, 2014

Alleged dates for iPhone 6 (and presumably iOS 8) release

The iPhone 6 release date is now being widely touted as 9 September 2014. It seems fairly likely that some differently sized devices will be released on or around that date. Whether or not the rumours about such devices being specifically sized 4.7" and 5.5" remains to be seen-- as will whether such devices actually turn out to be telephones or something more like versions of the iPad. Or put another way, is the ability to test your apps on an "iPad" of arbitrary size in Xcode a red herring...?

What this does means is that as developers, we have a schedule in place for testing and finishing development of new apps to take advantage of the more "iconic" of the new iOS and Mac OS features that Apple will probably be pushing. If you want to be among the first to take advantage of the new app extensions or continuity features, you have a month or so left! Now is also a good time to start testing your existing apps to see how they will cope with arbitrary screen sizes to make sure that-- as and when the newly sized devices actually appear-- it won't be too much effort to complete the process and get new versions of your apps out quickly if necessary...

Wednesday, June 18, 2014

New proposal for a Java bridge to native APIs

A new Java interface to native APIs, dubbed Project Panama, has been tentatively proposed. At this stage, it's hard to tell if it will go anywhere. What is true is that so far, the Java Native Interface as always been something of a second-class citizen. Although there are other projects such as Java Native Access designed to ease the pain of access to native APIs from Java,  a way to do so from within the standard libraries would surely be welcomed by many.

Friday, June 13, 2014

Is a split-screen interface really coming to the iPad?

You may recall a rumour that the iPad would soon be getting a split-screen interface, allegedly allowing two applications to run side by side in Landscape mode. The rumour has been received with skepticism and, given that WWDC2014 has now passed without a whiff of such a feature, appears so far to be untrue.

I actually assumed that the sources behind the rumour were actually referring to the new App Extensions framework-- whereby an application can offer up certain types if "plugin" to other applications-- that will indeed be introduced in iOS 8.

However, it appears that the rumour department of 9to5mac are persistent with their story. They cite apparent evidence in the form of a de-compiled section of code from the beta development kit which includes references to calls that apparently are designed to support a "multiple app display layout" of some sort.

My thoughts:
  • the de-complied code presented isn't exactly a smoking gun; it's not terribly clear exactly under what circumstances this code is intended to be used;
  • given that this year's update is highly developer-focussed, and that a split-screen feature could have significant implications for developers, it's an odd feature to "sneak in" outside of the developer conference-- Apple have now spilt the beans on various major new features to the OS, so it seems slightly unnecessary for them to be so secretive over this specific feature if it is really in the pipeline any time soon;
  • with the App Extensions framework, Apple have clearly put some thought into how they want app-app interaction to work; it seems a little clunky for them to introduce a separate feature that simply lets two arbitrary apps sit side by side;
  • on the other hand, there is good evidence that Apple are planning to introduce devices with more diverse screen sizes, and a larger-sized device would make a split screen function a little more viable.

Which iOS innovations are most anticipated by users?

At the beginning of last week, Apple made some fairly major announcements for the forthcoming update to its operating systems in what was-- in the words of Apple's CEO-- "the mother of all releases for developers". And it's fair to say that the new features open up a range of potential for new types of applications for Mac-- and for iOS in particular-- that were previously not possible. At some point or other, a release packed with features for developers will hopefully translate into a release packed with new features and applications for consumers.

Certain of these features, such as the new Health application which was widely leaked even before the event, appear to have been latched on to by the media despite the fact that they are probably only relevant to a minority of developers. Whereas, for example, CloudKit-- which has huge implications for the ease with which developers can now create a whole class of scalable client-server applications with minimal investment-- appears to be a subject of salivation for developers whose implications have yet to be appreciated by other commentators.

But as tech heads, it's still helpful to try and get an ear to the ground when it comes to finding out what matters to users. One insight is provided by French site igen, which is currently running a poll among users on which feature they have been most interested by.

At present (out of just over 2,000 respondents), the front runners are features at the "business" end of the spectrum: Continuity and iCloud Drive (50% and 16% respectively of respondents selecting these as their preferred feature). Interactive notifications and innovations in the messaging system, despite their swagger-laden demos at the KeyNote, have so far raised less of an eyebrow (12% and 6%). HealthKit receives an extremely low level of interest (3%), perhaps because of a current lack of adoption among health providers in Europe compared to the US (if not because of the fact that data privacy is more firmly entrenched in French law and digital ethos). Unfortunately, the survey did not single out App Extensions, which will effectively enable whole new categories of applications to be developed for the two operating systems.

You might not have rushed to incorporate Continuity or iCloud Drive into your apps. But from one informal survey at least, it appears that if you do so, you will be responding to two of the most anticipated features among users.

Monday, June 2, 2014

Is iOS 8 really trying to "crush" DropBox and WhatsApp?

In a recent article on Wired, Marcus Wholsen apparently comes to the conclusion that the new features announced for iOS 8 are an attempt to "crush" or steal the territory of DropBox and other apps. He sums up Apple's attitude as:

"Don’t try having your own ideas and not play with us, because we will take them, we will do them better and we will crush you in the process."

before going on to write:

"The new iCloud Drive for file-sharing and syncing takes direct aim at Dropbox, which Steve Jobs famously wanted to buy. Even the new iCloud photo features are an open assault on Dropbox’s new app Carousel for uploading and storing pictures"

in addition to effectively describing the new features for Messages as an attack on WhatsApp.

I must admit that I find such conclusions slightly surprising. Or at least, I see the new features in a different light.

As I see things, Apple is not simply trying to steal territory from DropBox. Rather, they are creating a more level playing field. Moreover, they're creating a more homogenous playing field that is closer to what users want and expect on a device nowadays. Via the new Extensions API, they are making it easier for DropBox-- but also for any other cloud storage provider, including Apple themselves-- to become more integrated into a wide variety of apps (and indeed into the OS itself). Users that find it convenient to use DropBox will not only be free to go on using it, but will find it integrated into more  apps. Now that cloud storage is an everyday expectation rather than a novelty, a major issue with iOS as it stands is that certain applications tend to be coupled with storing documents in one particular storage provider. The new functionality in iOS8 promises to iron this out and make applications much more "storage agnostic".

It's true that the new features of the Message app are largely playing catchup to other messaging services such as WhatsApp. Features such as sharing one's location are now so basic that it's somewhat surprising that they weren't already included. (Actually, I hadn't even noticed that this feature wasn't-- I guess I just assumed it was there if ever I needed it.) However, I really wonder if the presence or lack of such a feature is what will switch people over from WhatsApp to iMessage rather than more fundamental things such as which messaging system one's friends and contacts are using...? WhatsApp is presumably free to go on adding innovations to its messaging system; meanwhile, Apple has now added to its system features that have become de facto standards.

Mac OS "Yosemite" and iOS focus on "continuity"

A key theme of the updates to Mac OS and iOS this year is what Apple is referring to as continuity: being able to start a task on one device such as an iPhone and then continue it on another device such as a Mac. This notion was demonstrated at the WWDC keynote with the task of writing an e-mail on one device and continuing it on another, or going to a web page in Safari on a Mac and then picking up the same page on an iPad.

In principle, this could be one of the biggest and most useful innovations to the Apple ecosystem, and certainly received something of a rapturous response from the WWDC audience. But the devil will be in the detail: how extensively is this rolled out across the built-in apps, and what support do developers have to build this functionality into their apps, and how easily? I look forward to seeing more details...!

Apple announces the latest version of Mac OS, "Yosemite"



Friday, May 2, 2014

Microsoft blurs the line on "ending support" for Windows XP





Wednesday, April 30, 2014

Steve Wozniak on data security and privacy

Speaking at the Apps World conference this February, these were Apple co-founder Steve Wozniak's comments on the current state of affairs regarding the security and privacy of our data, in particular given our increasing reliance on the "Cloud" for mundane tasks such as listening to music while walking the dog...

Internet Explorer vulnerability given a high profile

A vulnerability in Internet Explorer announced by Microsoft last week appears to have received a much higher profile reaction compared to other vulnerabilities, with even the UK and US governments getting in on the act to advise people to switch browsers. (Some might say that advice about data privacy is a bit rich coming from them...)

The specific vulnerability is reported to be the exploitability of Adobe Flash Player using a technique baptised "Heap Feng Shui". Reported, that is, apparently not by Microsoft, who have so far disclosed little information other than that they are still investigating the matter.

And this may be one of the reasons for the higher-profile reaction. Since the recent retirement of Windows XP, the world is now coming to terms with the reality of a vulnerability report effectively saying: "There's a serious security risk whose details we are not fully disclosing, which we do not promise to fix on 29% of computers in the wild".

And it's a reality they may need to get used to.

Tuesday, April 1, 2014

Historic source code: Apple II DOS

Those programmers into a bit of nostalgia may be interested in various early Apple documents made available on line by the Computer History Museum. Notably, these include the Apple II DOS source code, in M6502 assembler. The code is a reminder of just how hideous and painstaking it must have been to have to code something such as a disk operating system in what is reckoned to be a relatively short space of time.

Other hand-scribbled planning/specification documents released in the bundle will make programmers feel more relieved about the contents of their own notebooks.

For those wishing to pore even further over historic assembly language with that feeling of "How on Earth did that ever work?", you may also wish to take a look at the Spectrum ROM Disassembly (I have fond memories of having had this in book form at one stage: with the same feeling of disbelief!).

Sunday, February 16, 2014





Update: more reports of Apple refusing Flabby bird clones.

Saturday, February 15, 2014



Friday, February 14, 2014

Preserving iPhone battery life: the practical and the ridiculous





How we still suffer the consequences of security-less protocols


Thursday, February 13, 2014

Some eye-openers for independent developers


Steve Wozniak on patents


DRM and copyright law




Monday, February 10, 2014

A reminder that cryptography researchers are not immune from cryberattacks




Sunday, February 9, 2014

Flappy Bird: Opinions from indie developers

My current high score at Flappy Bird



Friday, January 10, 2014

On the history of how to waste CPU cycles

If you've ever sat in front of a Windows machine thinking "stop animating the fucking dog and just tell me where the fucking file is", it turns out that operating system developers' penchant for wasting as much computing power as possible on computationally spurious activities at the expense of making useful functions instantaneous has a longer history than you might think.

Wired Magazine has recently published a talk by Steve Jobs from way back in 1980 (courtesy of the Compter History Museum archives) where, even at that time, he expressed the opinion that computers essentially had the raw processing power to meet the needs of many users' tasks and that he envisaged a future where latent processing would be spent on what we might term "user friendliness". Of course, by today's standards, he was surely setting quite a low bar in terms of the tasks that users expected to complete: on a spectrum going from "text editing on a green screen monitor" to "editing HD video in real time", people's expectations were generally towards the former end of the scale... But three decades on, it's interesting to note that much of Jobs' projections are true: a modern OS dedicates much of its code base and resources to essentially UI sugar.

What apparently wasn't envisaged by Jobs in the early 1980s (and indeed by those designing many of the fundamental communications protocols that still underly much of the Internet) was the proportion of resources that would need to be dedicated to security. How times have changed...

Thursday, January 2, 2014

Playing the game dev game

For those wishing to get a flavour of what it's like battling in the computer game industry without actually developing a game, check out Game Dev Story. This mobile game gives players a flavour of hiring, firing and training developers and being subject to the whims and volatility of the console market.