Announcing Freerisk

Last week, Toby Segaran and I announced our new project Freerisk at the O'Reilly Etech conference, in a talk entitled "I Just Don't Trust You: How the Tech Community Can Reinvent Risk Ratings".

In a nutshell, we're building a semantic database for all credit risk related data that we can find, and offering an API for others to implement their own credit risk calculators and publish these ratings back into the system.

This will allow investors to collect a diverse set of credit ratings and select the ones most relevant to them, make it easier for experimenters to create new risk metrics, and avoids a number of the payment for service problems in the current credit rating industry.

There's more to come, and I'll share those announcements here.

For now, here's our talk at Etech

For those of you that understand the current problems in the credit rating industry, you may want to skip forward to about the half-way point where we present the solution.

this is old.

but oh so good.

Hadoop Reduce Value-Iterators are Flyweights

Hadoop reduce iterators are pretty broken, so realize that before you waste tons of time like I did. The following won't work (never mind the crap job I'm doing protecting my heap):

List entries = new ArrayList();
while(values.hasNext()) {
entries.add(values.next());
}

If fails because Entry is a flyweight instance that gets recycled on each iteration. A nice optimization if you need it, but a rather heavy-handed assumption and one that should have been documentated, and an optimization that in this case leaves you with a  List populated with references to the same object instance. This could take you a while to track down.

If you, like me, can afford to cache your values in your reduce step and have a multi-pass algorithm which requires you to cache do something like this:

List entries = new ArrayList();
while(values.hasNext()) {
entries.add(new Entry(values.next()));
}

where Entry(entry) is a copy-constructor (you could use clone() if you've done nothing sexy to your class and for some reason hate portable design standards).

Hadoop is a great project, but they really drop the ball on documenting oddities like this.

Podcaster-gate

I'm impressed by the anger directed at Apple for Podcaster-gate. Good community backlash is always healthy. If you haven't heard about all this you can read more about it here to catch up, but in short, Apple didn't allow an application that competed with their iTunes application onto the iPhone, even though it didn't violate any of the provisions of the license agreement.

However, I'm quite unclear, why anyone professes shock or surprise at the situation.

It shouldn't be surprising that this happened, though there are many reasons for consumers to be upset by this. I'm sure Podcaster is a wonderful product that does many things that can't be done yet on the iPhone. We'll all miss out on those. But Podcaster wasn't ever going to make much money on this or ever reach scale because the strategy was naive and flawed; and so in the end, Apple did it's developer, Alex Sokiryansky, a favor.

Don't try to play nice with a bully

The strategy was naive because it assumed that Apple wouldn't be petty and protective of its turf. Past performance is no guarantee of future results but with Apple it's pretty a good estimate. Apple has a long history of screwing up independent software vendors' businesses and products by replacing them with built-in features. Konfabulator (now Yahoo! Widgets) and Watson are shining examples of wonderful, well-loved, products still trying to find the knife Apple stabbed them in the back with. Assuming your product will be treated differently isn't rational. The very fact that Apple wanted control over the App Store should have caused the cautious to flee the platform as fast as possible.

Don't take on your platform providers when they write the rules

Furthermore, the strategy was flawed because Apple had to react aggressively towards this product--Apple isn't just in the business of producing iPhones, iPods, and computers; they are in the business of owning the distribution of video and audio content, which lets them command premium pricing on those products (yes, yes, the design is a differentiator, too). Putting a product like this on their platform threatens the network effect that Apple is trying to build between their products--it threatens almost everything that has differentiated Apple circa 1997 from Apple circa 2008. These are the network effects that keep us coming back to them to buy their products--and cause us to deal with lesser products like Apple TV. Apple can't allow simple content distribution on their products because it can't allow the value they dominate to go away. This may or may not be best for consumers, but it's certainly best for Apple.

Don't assume you and your platform provider have the same goals

Finally, within the community, there's an argument that the process of accepting applications after they're done is problematic because independent software vendors cannot know ahead of time if their product will be placed in the App Store, forcing them to invest with no guarantee of an ROI. That is, of course, true. Because of this uncertainty, companies will need to work on products with a larger payoff than before, earning the higher risk these products take on. But the independent developer's product is Apple's good fortune. Who wouldn't want the natural quality filter that these products carry? Quality, not quantity, is a very Apple-like notion that this policy underscores precisely. If you don't help tell that story, they won't help you sell your product. We learned that from this fiasco.

Don't confuse forums for throwing sheep* with real platforms

There's also a question of what type of platform this is: are we looking at a platform designed to create real value for users, or are we supposed to building sheep throwing applications for it, to increase stickiness and virality for Apple. Despite assertions from Apple, it may be, and appears to be so far, that the iPhone will never be a platform to base your company on, but rather a platform in which you can distribute complimentary products to your core offering; products that grow your own strength while not threatening, or in fact complementing, Apple's own offerings. There's substantial financial evidence of this. Apple's iFund for creating new companies around iPhone applications is deploying capital at a slower rate than many expected, yielding only five investments so far, each of which is looking to harness the iPhone to access value from a market Apple doesn't have a hand in.

Net lesson: Apple owns this platform, and you should act like a good guest wanting another invitation, rather than than trying to grab all the attention for yourself.

The good news is there's a pretty interesting new platform coming out shortly that should be more than happy to host Podcaster.

* Tim O'Reilly first used this pejorative at last week's Web 2.0 Expo keynote.

Perhaps Denmark shouldn't outsource it's cultural exchange to these jokers

But I'm available for the right price.

Ostensibly a tribute to Olafur Eliasson.



Contains expletive laden substance.

Hat tip the quiet quiet

blogging at O'Reilly Radar for a bit

I'll be guest blogging on the O'Reilly Radar blog for a little while.

You can see my first post here.

Is language a function of your networking?

David Gravel is quoted as thinking so given the massive efficiency to shared languages in a global world.  It's a tidy and nice argument.

If you picked two people at random off the face of the earth and asked them to pick one language in which to communicate with someone they knew nothing about, which language would each person choose? The language they’d pick would depend on a series of “reciprocal expectations” — best guesses not just about which language you suppose the other person speaks but which language he thinks you suppose he will speak — which depends, in turn, on which one you think he thinks you suppose he will speak. And so on, until your head swims.

In today’s globalizing world, the probability is increasing that two random people would choose English for their best chance at unplanned linguistic coordination. And this isn’t merely a thought experiment: it’s being played out, with more information among the parties, in the decisions of hundreds of millions of people now learning English as a second language.

It reminds me a bit of David Foster Wallace's wonderful argument that grammar, too, is a function of your social setting (the pre-SNA way of saying networking).

You'll need a Harper's Subscription to view the link, but you should have one of those anyway.

These arguments make me wonder if, given this powerful network effect protecting the English language, there's any need to mandate a single language in the U.S., Britain. I'm also wondering just how the Quebecois intend to maintain French over time.