Writing to say I won’t be writing this weekend. I’ve got my head buried in putting together a proposal for what I think Etsy can be in 3-5 years. It’s incredibly fun stuff, and I’m hoping I can get the rest of the company as excited about it as I am. In the mean time, you can follow me on Twitter (as usual) or on Google Reader (as usual). Also, I’m almost done with Clay Shirky’s book Here Comes Everybody - it’s incredible; I definitely recommend it.

Hi everyone,

Bre Pettis has invited me to present at the next Ignite NYC event. You can read the NYTimes writeup here. For Seattle folks, this is basically a clone of the Ignite Seattle event, which Bre can pull off in good conscience since he was one of the founders of Ignite Seattle.

So I need to give a presentation about something. The presentation format is 20 slides in 5 minutes, with slides automatically changing every 15 seconds.

Please post a comment with your suggestion! I’m considering talking about the importance of long tail websites having sharing tools like the ones found in GReader and Tumblr and The Hype Machine, combined with platforms that allow clumping (aggregation) to happen like Meetup or Amazon Tag Communities. But maybe that’s old news? Or boring?

Thanks!
Dave

p.s. I almost used this photo instead of the one above.

Tags: ignite, presentation

I saw this post about IMDB as I was perusing Techmeme last night. Wow, talk about a tough situation - the new Batman movie gets voted up to #1 movie of all time even though it clearly is not deserving. Similarly, the long standing #1 - The Godfather - actually is getting voted down. What can you do? The community has been taken over by biased participants and agitators.

The holy grail is to build the perfect reputation system, where you count votes in relation to the voters “reputation”, but that is impossible to define. A reasonable alternative might be to take a cue from Wikipedia and lock voting for the Dark Knight (and possibly the top 5 movies). When Wikipedia articles are getting vandalized or constantly being fought over by various factions, moderators can come in and lock an article until emotions cool. Considering that movies have a strong temporal aspect to them - how many people will be talking about this summer’s blockbuster over Thanksgiving dinner - locking voting on the movie might do the trick. If the movie is unlocked after several months, truly passionate fans can go back and vote for their favorite movie, but the mob energy will be much more difficult to recreate. As to fixing the current vote tallies, I’m not sure. Reverting votes could result in an even more significant and wide spread backlash from the community. My recommendation would be to leave votes as they are and count on your loyal fanbase to reset the rankings to their appropriate levels. Good luck, IMDB.

Mine is the 21,120,387th visit to Coldplay’s MySpace page. I am not greeted warmly. The British band — which is known for giant pop hits, a sheen of fakery and the marriage of its lead singer to Gwyneth Paltrow — does not exactly rush out to greet me. The page is rudimentary and indifferently decorated, like the apartment of four couchbound soccer addicts who barely look up when a girlfriend comes in.

To get back to Josh’s article, I think he didn’t go far enough. Yes, individual’s behavior does change in response to technology, but what if we think about it generationally. Compare children today vs. people born in the 1950s or 1930s. They have a technological intuition that is astounding compared to their parents or grandparents. How is that happening? Now, I’m not arguing that there is some genetically detectable evolution going on with regards to the cognition of technology. I do think there is a behavioral co-evolution (in the same sense that Josh intends) where technology creates a generation of children who intuitively grasp that technology, and as that generation ages will produce even more advanced technology (leading to yet another generation of children more intuitive than the previous).

Where does that leave us? I’m not sure, but I do look forward to an ever-increasing rate of radical innovation in technology.

Tags: co-evolution, science, technology

Thanks to a Facebook Application called Friend Wheel, I can generate the visualization pictured above of my 549 Facebook friends (and still growing). It’s kinda fun to look at; my friends are listing around the edges of the circle, and a line connects to people who are also friends of each other on facebook.  The reds, oranges, and yellows are high school friends. The deep blues are Amazon.com friends. The greens and aquas and most of the rest are college friends.

I had dinner with my friend Steve McNally last night, who is roommates with my other friend Jake Tuck. Lisa asked me which one was I closer to. My response was that I had more history with Jake (we were housemates all through college, whereas Steve only lived in my house for half of college) but was probably closer to Steve since we shared a passion for baseball. Tough call, since Jake is a musician (as I am). Then Lisa asked me if they were friends with Will Paul. I said no, because Will is a hometown friend while Jake and Steve were college friends. So that got me into thinking about how to visual social networks and how inadequate two dimensions is.

Let’s try three dimensions. For the x- and y-axis, imagine an ideaspace - this is a plane that maps out the various interests people have, the hobbies they participate in, the fields they work in. So you have one circle for the friends you go to jazz concerts with, one circle for your photowalking friends, one for your baseball friends. The size of the circle is the number of mutual friends you have who share that interest. At the center, (0,0), is you. The circles in the plane are arranged such that the interests that are most passionate to you are closest to the center. Does this make sense? Two dimensional graph containing overlapping circles of various sizes, with the ones closest to center being of the most interest to you. Got it? Good.

Now for the third dimension, which is time. Over time, you will naturally transition environments. High school, college, work, living abroad, joining the local book club, marrying your spouse and meeting her friends and family, moving to the suburbs to raise a family, etc. Each of these events expands your social network and can form dense clumps. The third dimension in our visualization allows for the stacking of these clumps. It is more uncommon for connections to span the clumps, but it can happen and can be enlightening. I think seeing such a visualization would tell a lot about a person - what their interests are, who their friends are, and how have they changed over time. What’s your social network look like in three dimensions?

I haven’t blogged since the day Lisa came home from Italy. I think it’s because I’ve structured my life to be dichotomous - there are work hours (9 - 7), and then there are non-work hours. So when Lisa was in Italy, I was spending my non-work hours living my “online life” - blogging, reading blogs, sharing posts, keeping up on twitter, reading about friends via Facebook, tracking Etsy using Summize. Now that Lisa is back, I have almost completely replaced those activities with spending time with Lisa, which has been wonderful. I get home from work and dive right into the life we have together.

But it got me thinking - shouldn’t I be able to balance work, life, and online life? How can I fit in my online life without feeling like I’m ignoring the person in my home in favor of a virtual internet community? Just to be upfront, Lisa has been encouraging me to take time and blog and read and do all the other things she knows I like to do. I guess I just have my priorities straight - I like her better than the internet.

So I’m making an effort to come back to my online life, which should make Mark Blumberg happy (a friend of mine from high school who encourages me to blog, not this guy). I’ll fit it in before work or before bed, and we’ll see how it goes. I should probably declare bankruptcy on my Google Reader - it’s been a month since I’ve done more than skim headlines.

OK, time for a real post.

Four hundred and twenty eight posts ago, I started a thread on the Etsy forums. The thread was intended to have two purposes. One was to simply get a sense of the community and introduce myself to them. Second, I wanted to get some better intuition as to how sophisticated Etsy sellers (the majority of forum posters) are about running their business and the e-commerce business generally. So I asked a basic prioritization question - would you rather us make search better or fix a bug that would occasionally reset the page views counter on your item listing pages? (For those curious, the views system is stored entirely in a cache, and when the cache gets full and a record gets evicted, the page view number resets to 0. Clearly, the system was not engineered to be used in this manner.) My follow up was, if fixing the view system is not a priority, would you rather we get rid of it entirely or keep it broken.

The danger is to get lulled into an urgency to please. When hundreds of users are demanding a feature, you may feel compelled to acquiesce and build the requested feature. Before you do that, stop and consider Henry Ford: “If I did what people said they wanted, I would have built a faster horse.” (or something like that.) Customers are excellent gauges as when something is wrong, but can be extremely misleading about both what exactly is wrong and how it should be fixed. Furthermore, customers do not (or should not) have better visibility than you do into strategic goals, key business metrics, engineering resources, etc. They don’t have your long-term vision nor your understanding of complex dependencies. So don’t jump the gun. Listen, follow the comments to the source, and solve the root of the problem.

I considered Tim O’Reilly’s suggestion of investing in pieces of an “Internet Operating System”, which could be the answer although they’d have to fight Amazon, Google, Cisco and others for the bragging rights. Requirement (1) no, (2) yes.

I considered gaming, hardware, healthcare, social networking, and others…. but Microsoft is always involved there with mixed success. Is there anything left?

I’ve got a crazy suggestion. And yes, your own personal blog is the perfect place for crazy suggestions. So here it is - Microsoft should work to be the #1 destination site for vertical searching of the “organic web” (I just made that phrase up). The organic web would be defined as information that is continually changing. One example is airline ticket prices. Another is real estate, and another is classifieds. Microsoft should go out and develop / acquire any company who currently has the following properties: (1) The relevant data changes continuously, (2) The site is a leading player in their vertical, and (3) search is the main user activity on the site. Examples I can think of are Farecast (they bought this one), Craigslist (good luck there), and Redfin. With insider access to the data, Microsoft could provide superior search experiences to Google. Microsoft could then create a search portal that would be the first place everyone would go to search for data in these areas. Google’s crawlers can only go so fast - if Microsoft could provide a “real-time” search engine customized to a particular vertical, they could differentiate themselves in a very powerful way.

That’s it, keeping it short tonight since I’ve got a meeting in 10 hours with the CEO, COO, a few others.

So on to the technology. Here’s the clue that we’ll start with:

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practices that are more appropriate to a content management system.

From Twitter’s post on architecture and the problems they are facing 

When I read “content management system”, I’m thinking “blogging platform”. My guess is that Twitter is built to be a massively multi-user blogging and blog reading system - every user gets a blog to publish posts with and a blog reader to aggregate the posts of their friends. Considering Evan Williams was the founder of Blogger, I think it’s pretty reasonable.

So if you think of it that way, then the obvious way to architect the system is publishing via RSS and aggregating via RSS. When you write a new tweet, your message gets stored in the database. (Yes, shoving all of that data into a database is a really difficult engineering problem in itself. Assuredly they will partition across multiple databases if they don’t already.) The massive pain comes in when pulling in what your friends’ tweets are. Let’s talk through how it works. Your twitter homepage is acting like an RSS reader, so first it will lookup all of the feeds it needs to check - all of the people you follow. Then, for every person you follow, an RSS feed will be read or generated. The resulting set of RSS feeds will be merged back together and sorted chronologically. The result is your Twitter homepage.

Notice here that this is what is called a “pull” or “poll” model - you are checking for new posts whether there are new posts or not. This can generate a ton of unnecessary load on servers and databases, not to mention network traffic costs. With the advent of Twitter applications, these applications are constantly polling Twitter to see if there is anything new to publish. Ping, ping, ping. All to see if there is something new afoot.

Which brings us around to pricing. It is not, as Om suggested, Scoble’s fault for having 25,000 people following him. The cost is not sending one of his messages 25,000 times. No, actually it’s Scoble’s fault for following 21,000 people and constantly checking for new tweets from those people. It’s also the fault of power users like him using applications that aggressively use the Twitter API to check for new tweets - most likely the same people who use those applications are following large numbers of people.

As with all scaling problems, the first idea is “cache more!”. And sure, you can cache the heavy Twitter producers. But Scoble isn’t following just the big twitter users - he’s following everyone he can, because that’s how he believes he can get an edge on news and trends. Can the long tail be cached? Doubtful - there are too many users who fall into that category. Can you charge those who follow more than, say, 1000 people? Maybe $10 a month for every thousand people you follow, with the first 1,000 free? That could work, but it’s risky. Would Scoble, in the face of paying $210 a month, permanently switch to Pownce? Or Friendfeed if they built a twitter clone? How many would follow?

The solution, of course, is to do exactly what Twitter says they are doing - switch to a different model and scale horizontally (”throw more machines at it”). I’m interested to see how it turns out for them.