SkyGrid and the Emergence of Flow-Based Search

Burnham’s Beat:

GigaOm had a post today on a company called SkyGrid and its official company launch.  As an investor, advisor, and beta-user of the platform, I thought I would chime in with my own self-serving post mostly because I wanted to talk about the advanced technology and architecture behind SkyGrid and why it makes the company such an interesting case study in the evolution of search technology.

Simply put, SkyGrid represents a massive and exciting departure from traditional search architectures and technologies.  If I had to sum it up in a word, I would say that SkyGrid represents what I consider to be one of the first "flow based" search architectures, while traditional search engines are "crawl based" architectures.

Old Search: Crawl/Index/Query
While the technical
departure was necessitated by the leading edge demands of investment
professionals, it was these needs, and the lack of traditional search’s
ability to meet them, that exposed some of the most glaring weaknesses
of traditional search technology. Specially, traditional search
technology and architectures suffer from several glaring weaknesses:

Crawl-based: Current search architectures
collect information to index primarily by employing massive farms of
"crawlers" that systematically crawl IP address spaces. The benefit of
crawling is that it is exhaustive, the drawback is that it time
consuming and expensive.

One-off: Search platforms are designed around rapidly processing one off queries. This makes search engines highly useful and adept at finding "the needle in the haystack" but very cumbersome to use in situations where one just wants to get new results to the same old query.

Batch-based: Page rank and the other "secret sauce"algorithms behind most search engines today require a very expensive and complicated indexing process to be performed on "snap shots" of data. It can be days or even weeks before newly published content is crawled and properly…

---