Big Data: Confluence of technology and business analytics
Long, long ago men lead a nomadic life before gradually turning to agriculture. The invention of the wheel, fire and steam engine are often considered turning points in the evolution of mankind towards mechanization and increased life comforts.
Likewise the legendary Newton’s Law of Motion and gravity, Einstein’s Theory of Relativity that is now celebrating its 100th year or Law of Thermodynamics have all revolutionized science and influenced applied science. The invention of the computer, the arrival of personal computer and graphical user interface (GUI) all are milestones in the development in the digital era. It was the binary numbers zeroes and ones that is at the heart of assembly level languages .
Binary to Big Data
At the hardware level, zeroes and ones are powering the circuits in a computer, at the business level it’s the Big Data that is making a sea change in how companies devise marketing strategies to stay competitive. It could be composed of anything from single digits and multiple digits all containing vital information about the market, functioning of a machine, human body, ecommerce transactions or just about any day-to-day activity that may or may not have anything to do with buying or selling.
It is usual for businesses and accounting professionals to talk of assets and liabilities. Conventionally assets denoted the machinery, technology, know-how, human resources, infrastructure and also financial assets.
Now a paradigm shift is happening, along with these tangible assets, some pieces of single and multiple digits or data have become the most invaluable asset as organizations and markets grow in size. From the marketing and strategy point of view data has become the most important asset.
Businesses are growing in size and scale. No longer is small beautiful or viable. Multi country operations, big malls and large volume e-commerce businesses have set a new trend across the globe. To succeed in this big business data and data analysis have become critical. Businesses are after Big Data to utilize it gain market intelligence and understand customer requirements.
Confluence of technology and data analysis
Big data that organisations have will be meaningless unless there is a supporting technology to mine data, process and organize it for businesses to make use of this vital asset. Bernard Marr, renowned writer and analyst has said that companies irrespective of their size be it a Fortune 500 company or a small mom and pop store would require the use of Big Data witness the change it brings to businesses.
Big data is a collection of large datasets and they are in huge numbers that sophisticated programs are required to analyse and create meaningful information from them. It could be buying habits, frequency of going to movies, login frequency websites, online purchasing, ordering groceries, frequency of changing mobile handsets and so on.
Various tools, frameworks and techniques are used to analyse large data sets and they have become much sought after by the industry. According to experts, it is not the data that is important but what the company does with those data.
Among various technologies and platforms, Hadoop has emerged the most popular although it may have its draw backs. It is an open source development platform that is written on C, C++, Java and helps organisations analyse huge amount of data in real time.
Big data in real time
Collecting, storing, moving and analyzing is not a static activity but also a dynamic one involving real time environments. Data is being continuously collected for aeroplanes, automobile engines, monitors connected to patients in hospitals, online credit or debit card transactions all of which require sophisticated algorithms, programs, architecture and a robust in-memory processing capability.
John Schroeder, CEO of MapR said that they have Big Data applications that protect millions of American Express card holders from fraudulent transactions and in healthcare they are working towards providing improved treatment procedures for cancer patients.
Global IT majors such as Microsoft, Oracle, SAP, IBM are all on the cloud platform and also enabling solutions on big data.
Big Data and Internet of Things
Rapid changes in web and embedded technology has enabled a multitude of devices to be inter-connected to each other which is capable of sending data real time. An internet made of ‘things’ rather than people and computers has emerged.
Every piece of device we wear or use are capable of ending data which in turn would have wide ranging applications in marketing, design, health care among others.
Data Mining
Now, powerful super computers are deployed to mine data from relational databases and help statisticians and analysts to create models. Several innovators have come up with tools for development models for predictive analysis for better decision making by businesses. They also provide easy graphical user interface (GUI) and is very user friendly.
Career in Big Data
Naturally, enough the revolution in big data has spawned a whole new breed of experts who are associated with specific areas of this data analytics and technology. Among the technology skills hot in demand are Apache Hadoop, Apache Spark, NoSQL, machine learning and data mining, statistical and quantitative analysis, SQL, data visualization, data scientists, general purpose programming language skills. According to analysts opportunities are bound to increase in the next decade thanks to rapid developments in this area.
There indeed a huge demand for big data related expertise in 2015 with IBM having advertised 2,307 positions in the past twelve months in June, Forbes magazine said in a mid-year appraisal. The advertised salary for technical professionals with Big data training is $104,850. The most sought after skills were VMWare expertise, application development, open source technology, data warehousing and Python programming skills.
Industry wise, the topper in availing Big Data services are Professional, Scientific and Technical Services that account for 25% of the demand. Among other leading categories Information Technologies account for 17%, Manufacturing 15%, Finance and Insurance 9% and Retail Trade 8%.
Advantages of Big Data analytics
1) Storage, mining and analysis of data:
Big data technologies have enabled the deployment of both stored and real time data for a variety of business and mission critical applications
2) Market prediction & forecasting:
In the pre-big data era, companies were constrained to do meaningful data analysis real time or do predictive analysis in the absence of technology. Sample surveys and customer feed backs offered the only solution for strategists to innovate with new offerings to market.
3)Large amount of data is generated by businesses and in previous years, with in sufficient tools to collect and analyse them, businesses were failing to use an important asset with them.
4) In real time business environment, hacking and data theft can critically impact the working of an organization, the confidence of its customers and make it vulnerable to further attacks down the line. Big data and Hadoop have been proven to help organisations detect data theft. Data theft methodologies are evolving faster than anti-theft methodologies or prevention activities.
Is Big data the only requirement to succeed
The hype created by big data hasn’t gone off well with some critics who point out some of the problems associated with it deployment in industry. Some analysts have questioned whether there is a positive return on investment (RoI) and worth the time and effort taken to implement it in the first place. The second is with respect to the large volume of data and analysis which may not explain ‘why’ such consumer behavior is taking place.
Big data analysis can be effectively used in conjunction with traditional survey methodologies (thick data) that map the demographic patterns in saving, investment, buying and spending behavior across regions that gives a broader understanding of the market. Big Data may give a picture of what happened and how but ‘why’ it happens can be only understood by a broad understanding of the particular consumers or region based on demographic profile, life style preferenes, spending habits among others, according to skeptics of Big Data.
Major trends in Big Data technology
According to John Schroeder, CEO and Co-Founder of MapR, the company that delivers solutions on Big Data, had predicted the emerging trends for 2015 and most of them have appeared to be true.
Data Hubs to Data Lakes: Data lakes with scalable infrastructure seem to be favoured as they are economically attractive with a reduced per-terabyte cost).
Self Service: Self service big data will empower developers, data scientists and data analysts to conduct data exploration directly.
Data Agility
As database expands and more faster processing is required, the legacy systems seem to slow down the process. Legacy databases and warehouses have been found to be too slow and hence organizations are looking at how agile their data processing are.
Hadoop in innovation phase: Hadoop remains in innovation phase and Shroeder believes that more nuanced model of open source software combined with deep innovation and community development is possibly taking place.
Security Challenge
Big data storage and processing is now increasingly becoming vulnerable to security threats in the open source Hadoop system. However, the security features are yet match up with such threats and especially in comparison to more secure Enterprise Resource Planning (ERP) systems and relational data bases.
Cloud Computing
The rapid advances in cloud computing is enabling even small and medium enterprises to make use of SaaS (software as a Service), Platform as a Service (PaaS) and other platforms provided by vendors that enables them to utilize big data services at a much cheaper cost whereby costly licensing fees and installations are not required.
According to Bernard Marr, renowned author and analyst, sophisticated algorithms are deployed in the cloud space through SaaS that gives a more accurate picture of when, how and why a product is sold. Quoting Charlie Crocker of AutoDesk, he points out that until the arrival of Big Data customer feedback was a difficult exercise but with the sophisticated algorithms now at work, companies are better able to understand consumer behavior and create products for them.
The future of Big Data is bright
International Data Corp predicts big data market to grow at a compounded annual growth rate of 23% through 2019 with annual spending to reach $48.6 bn in 2019. IDC believes the three major sub markets: infrastructure, software and services will grow substantially over the next five years, with software –information management, discovery and analytics and application software-leading the charge with a CAGR of 26%.
IDC predicts services, including professional and support services for infrastructure and software, will grow at a CAGR of 22.7 percent. It forecasts that infrastructure — consisting of computing, networking, storage infrastructure and other datacenter infrastructure-like security — will grow at a CAGR of 21.7 percent and will account for roughly half of all spending through 2019.
“The ability to leverage big data and analytics to develop an integrated view of customer activities and business operations will provide competitive differentiation to companies across industries,” Jessica, Goepfert, program director for IDC’s Global Technology and Industry Research Organization, said recenty. “However, in addition to the huge opportunities, big data presents some significant risks and
Digital transformation (DX) will drive “everything that matters in IT” over the next several years. Succeeding in what IDC calls the DX economy means using technologies such as mobile, cloud, big data analytics, IoT, AI and robotics to “create competitive advantage through new offerings, new business models, and new customer, supplier, and distributor relationships,” according to Frank Gens, IDC’s chief analyst .
Key takeaways from IDC’s predictions
By 2020, almost 50% of IT budgets will be tied into DX (digital transformation) initiatives.
By 2018, Line of Business (LOB) executives will control 45%+ of all IT spending worldwide, over 60% in the U.S.
By 2017, over 50% of IT spending will be for new technologies (mobile, cloud, big data, etc.).
Even with rapidly evolving technologies and platforms, it is doubtful if all of the available data be analysed nor it would be required, some experts said. What is important is whether the relevant data is identified and analysed for the benefit of stake holders.
The post Big Data: Confluence of Technology & Business analytics appeared first on eduCBA.