Enterprises don’t need to store data to generate insights from streaming edge data, according to survey
Swim has announced new research that focuses on how and why enterprises are processing, analyzing, and benefiting from the increasing amount of streaming data. The survey, conducted in partnership with Virtual Intelligence Briefing (ViB) found that almost half (48%) of the participants continuously generate insights from streaming data in-stream without storing the data first, and that improving the customer experience is the number one business outcome their organizations expect from that approach.
The survey included over 200 IT professionals from a cross section of mid- to large-size companies, with more than half of them from companies with 5,000 employees or more. All of these organizations are using event-driven architectures, along with other streaming technologies. On average, the surveyed organizations use 3 different open source streaming technologies, with Apache Kafka being the most popular one used by 87%. Apache Pulsar is used by 17% of these organizations. Other popular open source projects used for streaming data solutions by these organizations include Apache Cassandra, Apache ActiveMQ and Apache Spark Streaming.
Today’s digital businesses and those moving towards digitization are rapidly embracing event-driven, in addition to historic batch-driven IT architectures. Event-enabling digital enterprises brings new capabilities, but also brings exponential growth in the streaming data volumes to be handled.
Key findings include:
– More than a third of organizations surveyed are building streaming applications – event-driven architectures are used on average for more than four purposes, with the most important being for data pipelines, messaging, microservices, data integration and stream processing.
– Custom-built streaming applications dominate – 70% of the surveyed organizations build their own custom streaming data solutions, while 30% choose to source commercial applications or cloud services when available. The majority of organizations (71%) building their own custom solutions are doing so by integrating custom logic with available infrastructure components, while the remainder use a pre-integrated commercial platform to benefit from consistency and reusability.
– Python and Java skills are most prevalent – When it comes to the technical skills used to develop streaming applications, Python (60%), Java (56%), and Kafka Stream/KStream (50%) took the lead among these organizations, followed by Javascript (39%), SQL (37%) and C#/C++ (35%).
“Customer perceptions and decisions, driven by a complex set of variables, are formed in real-time, in a competitive dynamic that changes moment to moment, from the unique buyer level, up to the market as a whole,” said Tom Riddle, director of research, Virtual Intelligence Briefing. “Creating the highest quality customer experience requires real-time, in-stream analysis of immense volumes of data, driving real-time adaptation. This process is democratized by Swim’s market-leading solution that greatly reduces the time, complexity, and resources required to build applications that create the most compelling customer experiences.”
“Streaming applications that continuously generate insights, drive actions and help to build always-on, situational awareness of the current state of the business are the new frontier for modern, data-driven organizations”, said Ramana Jonnala, CEO, Swim. “This research shows, among other learnings, how organizations are able to use massive, boundless flows of information to improve the customer experience by continuously processing and analyzing streaming data in-stream, in real-time, before persisting it to latency-prone data stores that impact their ability to promptly respond to and act on business-critical events.
Article Topics
Apache open source | applications | edge AI | Kafka | streaming data | Swim
Comments