Microsoft Launches New Big Data Stream Analytics Cloud Service

In the last couple of weeks we saw the fight heating up between Google and Amazon over big data in the cloud. Now Microsoft is calling the bet, announcing the general availability of its Azure Stream Analytics (ASA). The new cloud service, which was launched in public preview late last year, enables real-time stream processing of data from various sources, such as applications, devices, sensors, mobile phones or connected cars. In fact, Microsoft places strong emphasis on the use cases of the Internet of Things, a hot topic these days which Microsoft pioneered back in the 1990 but somehow managed to miss the wave, and is now trying to get back on it.

Microsoft puts emphasis, same as its competitors, on making its stream analytics service easy for development and operations, so small companies and even start-ups can get into this hot field without massive up-front investment. That while leveraging the power of the cloud to ensure transparent resilience and scalability, security and multi-tenancy.

Another interesting aspect is the built-in integration of Azure Stream Analytics with Microsoft’s Event Hubs, Microsoft’s Publish-Subscribe messaging service, which was made generally availability late last year, and is said to be able to log millions of events per second. Microsoft also targets this service for Internet of Things and telemetry ingestion use cases. This part of Microsoft’s offering is similar to Google’s Pub/Sub and Amazon’s Kinesis.

In a blog post by Joseph Sirosh, Corporate Vice President of Information Management & Machine Learning at Microsoft, he shares customer use cases by Fujitsu, NEC and Aerocrine. Quoting Allen Ganz, Director of Business Development at NEC:

NEC has found that using the Azure IoT Services has enabled us to quickly build compelling intelligent digital signage solutions that meet our customer’s needs and help them transform their business processes

Microsoft, same as its competitors, is aiming at providing a full and organic suite to cover the full cycle of big data ingestion, processing and analytics, to cater for the proliferation of big data use cases and ventures, and especially around the Internet of Things.

You can read more on the new Azure Stream Analytics here.

1311765722_picons03 Follow Dotan on Twitter!

Leave a comment

Filed under Big Data, Cloud, IoT

Google-Amazon Fight Over Big Data In The Cloud Is Heating Up

Google today announced that it’s releasing its Cloud Dataflow in open beta. This big data analytics service was launched in closed beta version at Google’s annual developer conference last June, with a major update last December when they released an open source Java SDK to make it easier for developers to integrate with the new service.

Just last month Google announced that it was moving its Cloud Pub/Sub into public beta. This service for real-time messaging is yet another layer in the overall big data and analytics suite that Google has been building up.

Google’s strategy aims to cater for the full big data and analytics cycle of Capture->Store->Process->Analyze from within Google Cloud Platform’s organic services (such as Pub/Sub, Dataflow and BigQuery), as well as with plugging in external popular frameworks such as Hadoop, Spark and Kafka, in a modular way.

Google Cloud Platform BI conf

Google Cloud Platform BI Suite

Google’s offering comes as a response to Amazon’s offering in the big data and analytics area, with services such as KinesisRedShift, Elastic MapReduce and Lambda. Interesting to note that last week at the AWS Summit in San Francisco Amazon announced Lambda service is generally available for production use. Amazon also maintains its smart strategy of tightening integration between their services, now enabling to run AWS Lambda functions in response to events in Amazon Cognito.

Amazon also puts emphasis on optimizing the infrastructure services for big data. A couple of weeks ago AWS launched new type of EC2 instances with high density storage optimized for storing and processing multi-terabyte data sets.

Another very interesting announcement from AWS last week was the announcement of Amazon Machine Learning new service, which gives an important dimension of analytics to their suite.

Amazon and Google are not the only players in the big data cloud services. With big companies such Oracle and Microsoft, this market definitely becomes hot.

1311765722_picons03 Follow Dotan on Twitter!

1 Comment

Filed under Big Data, Cloud, cloud deployment

Docker Raises $95M Series D Round

The buzz around Docker doesn’t cease. And Docker is riding the tide with a fourth round of funding, raising another $95 million, with massive $400 million valuation. Not that they need the money. Some say still have funding from the 2nd round. But they have a momentum, and they want to capitalize on it.

The $400M valuation is very high, but not surprising in the venture capital sphere, given that every major player out there is chasing to integrate with Docker, including giants such as Google and Microsoft. Docker is also storming into the cloud, establishing itself as the new model for application packaging, deployment and management in private and public cloud environments.

Docker has plans on addressing enterprise concerns such as storage, networking and security. It makes sense the current round would be used to push these topics. Now awaiting details from the source.

Leave a comment

Filed under Cloud, DevOps

Microsoft Onboards Docker To Provide Windows Containers On Azure Cloud

The trend is clear. Cloud and Docker are growing closer. And Microsoft isn’t staying behind. Last week Microsoft announced adding support for containers to its Windows Server and Azure cloud.

Microsoft’s interest in containers isn’t new. Last year Microsoft already officially added support for Docker containers on Linux VMs in its Azure cloud. Then, late last year it partnered with Docker to bring containers also to Windows Server instances.

Interesting to note that as part of its containers effort, Microsoft actively collaborated with the Docker community, including code contributions to the open source project. This is part of Microsoft’s ongoing efforts to expand open source involvement to gain foothold with emerging technologies.

Microsoft addresses the soft spots of the open source technology, aiming at needs of enterprises around security and hybrid IT. In a blog post by Windows Server GM Mike Neil he explains that:

heightened levels of trust may be required for enterprise systems or in hosted environments. Furthermore, developers often deploy into mixed operational environments where they may not have control of the platform where the application is deployed.

With the latest announcement Microsoft now offers, in addition to the Windows Server containers, also Hyper-V containers – containers built on top of Microsoft’s Hyper-V hypervisor technology. Hyper-V has been Microsoft’s response to VMware and KVM around “classic” virtualization, and Microsoft is now trying to leverage that to bring better isolation and security to containers. The challenge is to keep it efficient and avoid excess overhead of the multiple virtualization layers. We’ll have to see how well that works out.

The latest announcement also unveils the Nano Server, a slim version of Windows Server, optimized for running containers in the cloud. They promise minimal footprint in image sizes, deployment times, network bandwidth etc. Also important is that the technologies should be interchangeable, so that anything developed for one could be deployed also on the other without modifications. This will help supporting hybrid IT environments.

Microsoft will officially unveil and demo its new container offering end of this month at its BUILD developer conference. Then we’ll hear the verdict of the developer community.

1311765722_picons03 Follow Dotan on Twitter!

1 Comment

Filed under Cloud, DevOps

Big Data Processing on Amazon Cloud with New High Density Storage EC2 Instances

Are you running on Amazon’s cloud and struggling with handling big data? Then check this out: Amazon Web Services today released the new D2 series of EC2 instances, supporting dense storage which can handle multi-terabyte data sets.

The new instances do not only provide better CPU and memory spec. These instances are geared for sustained high rate of sequential disk I/O for access to extremely large data sets, which can get up to 3,500 MB/second read and 3,100 MB/second write performance on the largest instances. These new instances also have enhanced networking and support for placement groups which boosts access to remote instances. This makes the D2 instances classic for use cases such as Hadoop clusters and their MapReduce jobs, Massive Parallel Processing data warehouse, log processing etc.

It’s important to note that the disk I/O boost is for the local ephemeral storage, which is “gone” as the EC2 compute instance is “gone”. So it is up to the user to take care of redundancy of the data as needed, whether in RAID form, or using distributed file systems such as HDFS or GlusterFS. The new instances also come EBS-optimized by default, so you can offload local data as needed to EBS (Amazon’s native block storage) volumes while leveraging dedicated high bandwidth that doesn’t impact your regular network traffic.

Amazon guys did nice work integrating advanced features of the Linux kernel and of the Intel XEON CPUs. If you need to chew through your massive data sets, you’d want to check it out.

1311765722_picons03 Follow Dotan on Twitter!

1 Comment

Filed under Big Data, Cloud

Cloud and Docker Grow Closer To Bring Hybrid IT To Enterprises

Cloud computing and Linux containers (ala Docker and LXC) are two hot trends in the modern software delivery paradigm. They are both backed by strong and active global open communities, have rich tools, and new start up companies and projects are building their solutions with them.

But enterprises, while acknowledging the new paradigms, are still struggling with implementing them. One of the biggest concerns of enterprises is the vast infrastructure, which spans multiple systems, technologies, vendors, data centers, and even (private/public) clouds. Enterprises therefore require support for hybrid IT, with adequate automation to manage it. Gartner research shows that 75% of clients will have some type of hybrid strategy in place by the end of the year.

Puppet, the popular DevOps tool used for automating complex deployment environments, identified the need for provisioning and managing such mesh of cloud, containers and even bare-metal. This week on its latest Puppet Enterprise release it added support for AWS, Docker and bare metal.

IBM on its InterConnect 2015 conference this week announced an update to the container service beta on Bluemix, to provide capabilities such as push and pull containers from on-premises to off-premises service to support private & public clouds uniformly, and hosted private registry for container images for enterprise security and privacy.

The OpenStack community has been making efforts in the past few releases on integrating containers, initially as if they were virtual machine instances and later with full support so you can deploy containers through Heat orchestration just like you deploy applications and services. The Docker driver for Nova (OpenStack Compute), which has been developed out of tree so far, is expected to return to mainline in the upcoming ‘Kilo’ release next month.

Public cloud vendors staying behind either. Google adopted the technology internally, saying that “everything at Google runs in a container”, as well as developing Kubernetes open source project for orchestrating pods of Docker containers, and actively pushing it also into OpenStack via its recently announced partnership with Mirantis.

Amazon on its last re:invent conference announced its EC2 Container Service which lets you start, stop, query and manage container-enabled applications via API and using the rich AWS set of services.

VMware, which rules the traditional enterprise virtualization domain, made moves to adopt both open cloud and containers. First, it started getting actively involved in the communities and contributing code. Also, in the cloud front VMware launched an OpenStack-compatible version of its private cloud. On the containers front VMware partnered with Docker, Google and Pivotal to offer enterprises simplified path for containers over hybrid cloud model.

There are others exploring this field, such as RedHat, Cisco and even Microsoft, offering container integration in all levels from hardware through operating systems, hypervisors, cloud management systems, to orchestration and monitoring tools. We shall be seeing a growing number of such offerings of converged solutions for hybrid IT, more targeted at the complex environments of enterprises, and with enterprise-grade tooling and maturity levels.

1311765722_picons03 Follow Dotan on Twitter!


Filed under Cloud, DevOps

Amazon Acquires Internet of Things Startup 2lemetry

Amazon has quietly acquired the Internet of Things startup 2lemetry, which offers ThingFabric – a platform for the integration of connected devices across enterprises. Though no formal press releases clarified the details of the deal, sources at Amazon did confirm the deal to TechCrunch a few days ago. Amazon however did not share what it intends to do with the acquisition, and whether it indeed relates to 2lemetry’s IoT platform or to its retail beacon services or facial recognition technology.


The acquisition comes as no surprise, as 2lemetry has been exploring integration with Amazon’s cloud services (AWS) for a while now. Most recently 2lemetry’s engineering has been exploring how the AWS Lambda service works together with the MQTT IoT communications protocol (an OASIS standard) and 2lemetry’s ThingFabric. Another interesting direction for integration is with Amazon’s Kinesis big data stream processing cloud service, plugging in terabytes of data feeding in from hundreds of thousands of IoT sensors via 2lemetry’s platform.

Amazon has been trying to position its AWS cloud services with respect to the Internet of Things, with analytics services in the cloud or connected devices such as Amazon Echo. AWS mission statement says that

Amazon Web Services provides the services, security, and support to connect the Internet of Things on a global scale.

Amazon’s recent acquisition is probably a response to the moves by its public cloud competitors, most notably Google’s investment in the Internet of Things. Interesting to note that 2lemetry is part of the AllSeen Alliance open standards group for IoT. Amazon’s new arm in the AllSeen Alliance is a match to Google’s own arm in the Thread Group open standards group (via its Nest acquisition). Now awaiting Amazon’s and 2lemetry’s official statements of the joint path of the IoT in Amazon’s cloud.

1311765722_picons03 Follow Dotan on Twitter!

Leave a comment

Filed under Cloud, IoT