It doesn’t have to be that way, and this research note looks in detail at how components of technologies like Hadoop are finding their way into tools that target less-sophisticated users — from business users to receptionists to high school students. Thanks to cloud-based services, data visualization tools and more, analytics can be made easier, and maybe even fun.

The post Cloud computing and trickle-down analytics appeared first on Gigaom.

To make sense of the ever-growing amount of information that business and IT leaders must know about this space, GigaOM Pro has gathered six of its analysts to discuss several sectors of the current cloud market. It is our hope that the pieces in this report will help you, GigaOM Pro readers, to make the best decisions for your businesses when it comes to using the cloud.

As with any discussion of a market, this list of topics is not exhaustive, and we encourage you to weigh in with your own thoughts and ideas in the comments section.

The post Cloud computing infrastructure: 2012 and beyond appeared first on Gigaom.

One major solution to the big data skills shortage has been the advent of consulting and outsourcing firms specializing in deploying big data systems and developing the algorithms and applications companies need in order to actually derive value from their information. If McKinsey’s predictions hold true, these companies will continue to play a vital role in helping the greater corporate world make sense of the mountains of data they are collecting.

However, if the current wave of democratizing big data lives up to its ultimate potential, today’s consultants and outsourcers will have to find a way to keep a few steps ahead of the game in order to remain relevant.

The post Why service providers matter for the future of big data appeared first on Gigaom.

However, as more organizations get accustomed to Hadoop, they’re starting to realize its shortcomings. That has led to more attention for startups claiming easy analytics and real-time processing, two areas for which Hadoop isn’t really designed. Thankfully, the fourth quarter also brought news of innovation on the networking front that will be necessary to help big data workloads scale not just across servers, but also across geographies.

SaaS startups made out particularly well too, with venture capitalist and large vendor money flowing like water. Valuations for these companies are sky-high and getting higher; seeing mega-software vendors such as Oracle and SAP buying SaaS companies for billions won’t do anything to reverse the trend.

Down the stack from SaaS, the fourth quarter shed some light on the Infrastructure- as-a-Service (IaaS) space. We learned a little more about how much revenue Amazon Web Services might be producing, this despite it being far from perfect on the security front. We also saw how other providers might look to compete with AWS by building their own unique services that depart from what AWS does so successfully with its portfolio of on-demand services. These new efforts include everything from leveraging inter-data-center connectivity with other service providers to building a federated ecosystem of customers that freely exchange computing capacity among each other.

Overall, it was a bright quarter, marred somewhat by fallout Thailand flooding that closed hard-drive manufacturing facilities and sent prices for remaining units through the roof. Everyone from server makers to cloud providers was and continues to be affected because a lack of components for building servers means a lack of servers to scale out cloud infrastructures.

The post Infrastructure Q4: Big data gets bigger and SaaS startups shine appeared first on Gigaom.

It doesn’t have to be the one-sided affair it is today, in which companies have all the data and all the rights, and we shouldn’t have to be afraid of who’s doing what with our information. With laws, products, practices and education, data can become a far more valuable currency than cash ever was.

With that in mind, this research note examines five issues that must be addressed by policy makers and entrepreneurs so that they can deliver on our data-driven digital future.

Companies mentioned in this research note include Facebook, Twitter, Netflix, Yahoo, Foursquare, Dropbox, Verizon, EMC, Google and Apple.

The post Dissecting the data: 5 issues for our digital future appeared first on Gigaom.

Last quarter we highlighted the fast maturation of the Platform-as-a-Service and big data spaces. Those two trends only picked up speed during the third quarter of 2011. Joining them on the cusp of IT greatness, though, are the OpenStack project and flash storage.

For PaaS, the third quarter was all about expanding horizontally to cover more programming languages and bring in more developers. However, the more- experienced providers, such as Heroku, also started building upward to deliver specific offerings for specific application types such as Facebook and mobile.

In the big data world, it was much more money for Hadoop startups and even larger presences for mega vendors EMC and Oracle. The use cases for all varieties of big data analysis in several industries cleared up, too, with social-media data driving the ship in terms what companies want to analyze. NoSQL databases also matured significantly, and several companies now have the additional VC money to prove it.

But those stories have been getting louder for some time now. What shone brightest this quarter might well be the OpenStack cloud-computing project, which garnered serious validation via HP, Dell and Citrix, who are building their cloud strategies around the platform to some degree. New products from NASA cloud veterans don’t hurt the hype either, and OpenStack got two in the form of Nebula and Piston Cloud Computing.

The flash storage market saw less funding this quarter than it did in the last, but that was offset by a number of significant product launches and at least one major end user. Violin Memory, Fusion-io, Nutanix, Nimbus Data Systems and more rolled out new products, and they’re all citing consistent performance at price points comparable to similar hard-disk-drive-based products. This suggests that flash will finally overcome the high prices that have hindered adoption thus far.

Of course, the third quarter wasn’t all lollipops and rose petals. We saw new computing technologies and delivery models such as tablets wreak havoc on both HP and Cisco, and there are concerns (aren’t there always?) about how the Internet will handle our increased use of streaming video and cloud computing. Unfortunately for HP and Cisco, the latter problem might be an easier fix than the strategic woes facing them.

The post Infrastructure Q3: OpenStack and flash step into the spotlight appeared first on Gigaom.

Big data and Platform-as-a-Service offerings highlighted the second quarter, suggesting that we can expect to see a shift in enterprise IT practices around application development and analytics very soon.

On the PaaS front, we saw new projects like DotCloud and Cloud Foundry launch and gain incredible momentum in just a few short months. We also saw existing platforms such as Heroku, Google App Engine and Microsoft Windows Azure mature in some very meaningful ways. All of this means that developers — even of the enterprise variety — won’t be able to avoid PaaS for much longer. Pricing model aside, the capabilities will be too much to resist.

The big data activity was broader in scope, ranging from major new Hadoop vendors such as Hortonworks, EMC and MapR to heavy investment in flash storage that will speed the serving of data to processing engines. Even more interesting use cases for Hadoop and other big data tools popped up to demonstrate that with the right technology running the right algorithms, we can use data to power an endless variety of applications.

In other areas, we saw an uptick in cloud-computing plans from large vendors, with IBM, HP and Red Hat showing off very respectable cloud-computing products services. OpenStack also continued to mature and pick up both contributors and users, meaning that anyone selling wholly proprietary cloud software will have a formidable open-source alternative to worry about.

In the world of web infrastructure, Facebook caught our eye by launching an open- source project around the designs for its specialized servers and data centers. How this will ultimately affect the hardware and facility market remains to be seen, but important institutions such as leading service providers and banks are already coming together to try and figure that out.

It wasn’t all great news, though, as the second quarter kicked off with the weeklong Amazon Web Services outage. That event exposed architectural flaws on AWS’ side, as well as among a large number of its customers. Actually, the good name of cloud computing came out of the incident relatively untarnished, as the media and the companies involved focused on how to resolve the problems.

The post Infrastructure Q2: Big data and PaaS gain more momentum appeared first on Gigaom.

On a broad level, the current state of cloud computing across all sectors is that interest and plans are sky-high but actual usage still lags. There are notable exceptions, however, with regard to SaaS and cloud storage, because cloud-based email and backup services, for example, which are incredibly convenient, are broadly applicable and relatively risk-free. Where cloud computing is in use, it appears that big-name incumbents such as Amazon Web Services, Rackspace, Salesforce.com and Google still receive the lion’s share of users.

Whatever the adoption rates, though, cloud providers of all types have their foot on the gas pedal when it comes to innovation. From IaaS providers offering advanced hardware architectures and services to PaaS startups making entire application stacks literally a push-button experience, the cloud looks like a much more compelling place to host one’s applications with each passing day.

No company trying to sell cloud computing can afford to be lax on innovation, either. The rapidly growing numbers of providers in areas such as SaaS and private clouds — and the huge amounts of venture capital pouring in — indicate that competition is fierce. This incredible growth will ultimately lead to consolidation, but everyone must keep innovating until adoption picks up and true leaders (apart from today’s household names) emerge in these new sectors.

There are still some major concerns with the cloud, though, particularly around security and lock-in. A number of open-source projects and products are now attempting to resolve the lock-in problem, but there’s still a long way to go. Security, too, is the focus of much talk and a fair amount of activity from industry organizations, but it looks as if it might take a damaging breach to kick-start an industry-wide approach to resolving some core issues.

Going forward, companies within all cloud computing sectors will continue to add new features and capabilities, and they will attempt to improve their palatability to large enterprises willing to spend lots of money on IT. On a broader scale, however, we will see improved integration across the various cloud computing sectors, as customers grow tired of their myriad cloud services operating like silos — one of the major problems with the legacy IT practices that cloud computing fundamentally seeks to change.

Assuming cloud adoption takes off as data suggests it will, cloud services integration will provide major opportunities for cloud providers and software vendors willing to open up their platforms to the greater cloud ecosystem, as well as for third parties hoping to play the middleman in helping to bring together many different services.

The report includes data from the Future of the Cloud survey conducted by North Bridge Venture Partners in conjunction with GigaOM Pro and the 451 Group, as well as numerous charts illustrating the concepts discussed.

Stacey Higginbotham provided additional writing and analysis.

The post A field guide to cloud computing: current trends, future opportunities appeared first on Gigaom.

Infrastructure as a service has been talked about, alternatively in hushed and gushing tones. Grid computing, utility computing, on-demand computing — they were all ways to describe what Amazon Web Services had delivered. Fast forward to today, when Amazon and others are moving at Internet speed, trying to offer better security, faster networking, more compliance and a host of other products that are attempting to meet the demands of startups, consumers and enterprises alike. It’s not perfect, as Amazon’s two-day outage earlier this year attests to, but it’s certainly good enough – and getting better.

We launched our Structure conference in 2008 because we saw the cloud-based infrastructure revolution was going to create new opportunities. As observers, we’ve talked to hundreds of people about cloud computing and its ecosystem. On our Structure channel, we cover the gear and software that comprises the cloud, the services and the people who are changing the industry. Now for the first time, we’ve decided to condense that knowledge into the Structure 50, a list of the 50 companies that are influencing how the cloud and infrastructure evolves. These are the ones to watch — at least in 2011. You’ve heard of some – such as Amazon or Dell. Others – such as Nicira or Boundary – are probably not yet on your radar. But they should be. All of these companies, big or small, have people, technology or strategies that will help shape the way the cloud market is developing and where it will eventually end up.

To the companies who made it on the list, congratulations. For others who missed out, in the future anything is possible. And for those who are still drawing their plans on a piece of paper, we are patiently waiting for you to change the world.

The post The Structure 50: The Top 50 Cloud Innovators appeared first on Gigaom.

In the case of IaaS, coming back down to Earth isn’t a bad thing; it’s just a sign that the market is settling into a state in which it likely will remain for some time to come. Amazon Web Services continues to lead in terms of customers and innovation — it released Elastic Beanstalk, CloudFormation, Dedicated Instances and an enhanced Virtual Private Cloud service during the quarter — while Rackspace, buoyed by momentum around OpenStack, will be its primary competitor for mainstream customers. As the IaaS market activity illustrates, however, there will always be a case for enterprise-focused providers as well as cloud providers connected to vendors like VMware and Microsoft and pushing highly integrated hybrid clouds.

The big data space, on the other hand, looks a lot like cloud computing did a few years ago: There are so many players and so many terms floating about, it’s difficult for outsiders to get a handle on who’s who and what’s what. For example, where there once was Cloudera offering a distribution of the Apache Hadoop project and some associated products and services, there now are more than a half dozen companies, all utilizing Hadoop in slightly different manners and making slightly different claims. The same goes for NoSQL databases, the proprietors of which still have some educating to do within traditional companies, but whom will have an even more difficult time doing so with the recent proliferation of projects, terms and well-funded startups. None of this is bad news, of course — this type of activity validates the value of the technologies — but it does lead to confusion as lay customers try to determine which “big data” tools are right for their needs.

The other truly noteworthy realization gleaned from the quarter is that Intel won’t be left to die if low-power servers based on x86 processors catch on like the buzz late last year suggests they will. The reason is that Intel is pushing its own line of low-power x86 processors in both Xeon and Atom forms, and even has designed a dual-core server-ready Atom processor for massively parallel startup SeaMicro. If it’s energy usage that’s driving the discussion around ARM processors, Intel sought to prove during the quarter that businesses don’t have to abandon the x86 architecture to achieve that end.

The post Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight appeared first on Gigaom.