Google announces Cloud Dataproc

Google has launched its Cloud Dataproc data storage and processing service that the company promises will make using Spark and Hadoop easier, faster and cheaper.

The managed service allows organisations to take advantage of open source data tools to improve batch processing, querying, streaming, and machine learning on Spark and Hadoop.

Automation makes controlling clusters a whole lot easier and with support for turning off clusters (and not paying for them) when you don't need to use them, you can save money too.

“Cloud Dataproc automation helps you create clusters quickly, manage them easily and save money by turning clusters off when you don’t need them,” Google Cloud Platform product manager James Malone wrote in a blog post. “With less time and money spent on administration, you can focus on your jobs and your data.”

Cloud Dataproc clusters are fully scalable and take around 90 seconds to set up compared to the usual five to 30 minutes. They can be shut down in the same time frame, making it an extremely efficient use of time for developers.

Additionally, Cloud Dataproc clusters can be used without the need for administrators because they are controlled directly from the Google Developers Console, the Google Cloud SDK, or the Cloud Dataproc REST API

The service will also fit into the rest of the architecture you're using in your organisation with built-i integration with services including BigQuery, Cloud Storage, Cloud Bigtable, Cloud Logging, and Cloud Monitoring.

Cloud Dataproc costs 1 cent (about 0.6p) per virtual CPU in a cluster an hour, plus the cost of other cloud resources you use. Companies are only charged for what they use, with minute-by-minute billing, with a ten-minute minimum charge.