The Wild West of Big Data

Print Email


It is very easy to compare the atmosphere of this year's Strata Conference and the big data ecosystem to the atmosphere of the Linux and open source ecosystem around the turn of the century.


(And yes, I get a little trill in my fingers when I get to type "turn of the century.")

The comparison is not literal: I made this observation to someone I met at here at Strata this week and he dug into me a bit on how his company's offerings were not related to open source technologies.

No, the comparison is more broad: around 2000 and 2001, the Linux community received a massive amount of business, media, and financial attention when companies like IBM announced they would be investing billions of dollars in Linux. Shows like LinuxWorld were suddenly populated with lots of corporate attendees, literally standing on the sidelines of the large open areas, wondering how the hell they were going to communicate with all those hackers slumped in the beanbag chairs.

There are beanbag chairs here at Strata (over in the MongoDB booth), but every time I walked by, no one was sitting in them. The traffic on the show floor was heavy, and the atmosphere was very much the same as you would find at any corporate-oriented trade show: lots of attractive young men and women staffing booths and talking up their products.

There's an energy here, in the sessions and in the halls, as thousands of attendees try to figure out how they can catch on to this rising star known as big data. The excitement goes both ways: the job board is full of papers looking for data scientists, and there are a lot of "I'm Hiring" ribbons stuck on the bottom edges of badges. Companies want to get in on this action, and they are desperately seeking talent to fill their ranks so they can start raking in the money.

Money is here, too. Sort of. There is a lot of hype in the air, and when you start talking about customer bases with companies on the floor, there's not a sense of a lot of cash floating around. Yet. Still, the potential for financial gains zings through the air like a static charge. Everyone here is shuffling their feet on the metaphorical big data carpet, hoping to touch the right idea and get that pop! of fame and fortune.

In these respects, history is repeating itself, and I actually find myself becoming a little melancholy for the days of Linux yore.

But even as the inevitable comparisons ensue, there is one thing decidedly missing from the Strata Conference, and indeed the larger data ecosystem: community.

Community is assuredly a part of the big data sector--Hadoop, Hive, Lucene, and Solr, to name a few, are all open source projects. But the actual presence of these communities is non-existent on the show floor and in sessions. A few Birds of the Feather sessions are dedicated to user groups, but that it all.

So really, the community isn't missing so much as invisible, and that raises a sharp difference between the Linux and open source sector of a decade ago and the big data sector of today. Community in big data is acknowledged as a corporate resource, and that is all. Commercial vendors here seem to pay little more than lip service to their open source origins, and usually only to mention how they contribute back to projects like the ones mentioned earlier.

The suits are now the ones sitting in the beanbag chairs.

To be fair, communities of commercial entities are communities, too. But there is no sense of the grass-roots, hacker-dominated communities that were so much a part of the Linux community's DNA. Here, it seems, that gene as been snipped away.

Nor is there the sense of a massive outside competitor looming over everyone's collective heads. There is no Microsoft or Apple sitting on the outside making potshots at big data every chance they get, as they did (and still do) with Linux.

Big data's community is purely commercial and without the threat of a big competitor to stand in its way. In this sense, it is more of a gold rush than Linux ever was, because without the checks of the internal and external pressures that the early Linux community endured, there seems to be nothing that can get in big data's way.

Which may be why we are seeing, even now, signs from experts that are warning potential customers and the vendors willing to take their money to slow down and start really thinking about what they want to do. It was the theme of a few talks here at Strata this week, which were a strong contrast to the lookiloo, see-what-you-can-do talks that dominated the rest of the agenda.

This caution is a welcome thing. The potential benefits of big data are real, but they are not universal, and therefore need to be applied with care. The Wild West strike-it-rich fever is still upon attendees at Strata, and the lessons of those times and other more recent booms need to be learned before the halls of Strata become like ghost towns.



Company Information



+31 (0)70 737 0353


+31 (0)70 737 0469


This email address is being protected from spambots. You need JavaScript enabled to view it.

Visit us:

Vuurtorenweg 35
2583 XL,  the Hague
The Netherlands

All about Grey Hippo

Anything in here will be replaced on browsers that support the canvas element