A comprehensive guide to data analytics

What is data analytics?

We all know that a company’s data asset is its most valuable commodity. In fact, data collection is the gold rush of this generation. However, at best, raw data is not useful. At worst, it impedes mission-critical decision-making across an organization. Enter the sifting pan: data analytics. 

 

Data analytics is the collection of techniques which process raw data to draw conclusions and gain insights from them. In business, this usually means revealing trends and behavior which can be used to improve operations. 

 

Traditional, manual methods of extracting the right information have since been written into software algorithms and then automated over subsequent decades. With increasing computational power and newly-developed techniques, data analytics has come to encompass a wide range of processes.

 

The four types of data analytics with examples

 

Data analytics is structured in four layers, with each subsequent layer building upon the last.

 

Descriptive analytics

 

Descriptive analytics is the simplest type of data analytics and forms the basis of the other types - simply ‘describing’ what has happened. Descriptive analytics doesn’t deal with why changes have occurred in the data, or with cause-and-effect relationships.

 

An example of descriptive analytics would be historical transaction data collected over several that could be analyzed to show that sales of a product tend to peak during October, November, and December. This trend can be shown visually in a bar chart or line graph.

​

Diagnostic analytics

 

Diagnostic analytics uses data to explain why changes have occurred, and establishes causal relationships. So for instance, you could compare concurrent trends and discover correlations between variables. For example, increased sales during October, November, and December are explained by their correlation with the festive shopping period, when customers are more likely to buy gifts.

 

In many use-cases, diagnostic analytics can be split into two stages: ‘discoveries and alerts’, and ‘query and drill-downs’. ‘Discoveries’ process the results of useful current statistics, such as finding the app with the most downloads, while ‘alerts’ warn of potential issues before they occur, such as increased server downtime which may impact online transactions. Queries and drill-downs examine past data in more granular detail, such as showing that expected sales numbers dropped due to an infrastructure disruption.

 

Predictive analytics

 

Predictive analytics use data to forecast future trends and events. Just as diagnostic analytics builds upon descriptive analytics, predictive analytics uses the correlations identified in diagnostic analytics to create statistical models which make useful predictions. These predictions help organizations to formulate strategies based on likely scenarios.

 

For example, given the sizes of the surges in sales that have occurred during the run-up to the festive period over several years, predictive analytics forecasts the volume of the same surge next year.

​

Prescriptive analytics

 

Whereas predictive analytics predicts which outcome is the most likely, prescriptive analytics estimates the impact of your actions on the outcome - i.e. it measures the incremental effect of an intervention - and recommends the action that is most likely to produce a favorable result. Prescriptive analytics often combines AI (usually ML) with Big Data to produce the most accurate results.

 

Prescriptive analytics consists of optimization and random testing, which are used to try out a range of interventions to see how they affect the forecasted scenarios, and determine which ones produce the highest chance of positive outcomes.

For example, given the sales volume predicted during October, November, and December, prescriptive analytics can forecast the website load, number of customer service agents, and quantity of stock required. Armed with this information in advance of the season, teams can plan to ensure they can  cope with the extra demand.

​

​

The benefits of data analytics

 

The advantages of data analytics are many-fold. Here are some of the main benefits.

 

Increased revenue

 

Customer data, such as email addresses, past purchases and demographics, can be gathered from multiple channels, including e-commerce, in-store purchase surveys, and social media. The data sets can then be used to build detailed customer profiles which enable businesses to offer more personalized experiences. For example, by analyzing buying habits, an online store can display products which may be of interest to the user when they log in.

 

Additionally, by collecting and analyzing customer data effectively, companies are able to recognize behaviors, which enables them to discover new audiences as well as better target existing customers. This type of data analytics is heavily used in behavior-targeted advertising. Altogether, these additional income streams can dramatically boost total revenue.

 

More efficient operations

 

Collecting and analyzing data about everyday operations can help bring insights into inefficiencies, or suggest alternative ways of running things. For example, employee efficiency can be monitored over time to identify the best times of day for certain tasks. 

 

Data analytics can also be applied to the supply chain. For example, a company can identify which supplier is the most reliable, and then make adjustments to use them more heavily when required. Predictive analytics can be used to predict delays and identify productivity bottlenecks.

​

Better decision-making

 

Before committing to an important decision, predictive analytics can be used to forecast likely outcomes in given scenarios, so that teams can choose the most beneficial path. It can then be used to suggest the best way of dealing with these outcomes. For example, statistical and analytical models can determine how changes to pricing and product lines are likely to affect revenue.

​

Reduced risk

 

Companies face many different types of risk. These range from financial losses, such as uncollected or bad debts, fraud, and theft, to legal and reputational damage, such as customer health liability and employee safety. Of course, you can put preventative and restorative measures in place, but data analytics can help teams better understand these risks and find better ways of dealing with them. For example, banks use modeling to detect suspicious payment card transactions in real time, and alert the appropriate cardholders.

 

Data analysis vs. data analytics

 

The terms ‘data analysis’ and ‘data analytics’ are often confused. While their meanings are similar, they do have important distinctions. ‘Data analysis’ is the detailed examination of the elements or structure of data, whereas ‘data analytics’ is the systematic computational analysis of data. Therefore, data analysis can be thought of as a subset of data analytics, which more widely involves the manipulation of data and the prediction of outcomes to aid in making decisions.

​
 

Data science, data analytics, and data engineering

 

The terms ‘data science’, ‘data analytics’, and ‘data engineering’ are often used interchangeably. But while activities in these fields can overlap, the definitions of the roles are quite different.

​

 

The difference between data scientists, data analysts, and data engineers

 

Data engineers create data infrastructure, such as building pipelines to gather data from users’ devices and send it to servers. Essentially, data engineering deals with the tasks that collect data into a database, and requires detailed knowledge of APIs. This includes figuring out how to merge data from disparate sources, optimizing data pipelines and storage for cost-effectiveness, and ensuring that data is accountable, validated, and correct.

​

Meanwhile, data scientists coconsume the data from the structures built by data engineers. They use statistics to find patterns in the data which can then be used to make predictive models. This often involves the use of ML and deep learning. The main role of a data scientist is to build tools that solve problems by using data.

 

Finally, data analysts use the tools built by data scientists to process and interpret new data. This produces actionable insights to help drive a business forward. Usually, data analysts deal with structured data sets once they have been generated from unstructured data sets by data engineers and data scientists. Data analysts typically use static modeling techniques to summarize results, rather than the dynamic techniques used by data scientists.

 

An example of data science

 

Data science is often applied to improve logistical efficiency, such as in the freight industry. For instance, using past data from drop-off and deliveries, ML can be used to discover how delivery trucks can be rerouted to avoid traffic jams or weather disruptions. By simulating outcomes from a number of different workarounds, and choosing the best available results, this type of model can save delivery companies precious time and cost.

​

An example of data engineering

​

One typical data engineering project is data ingestion from different sources into a cloud database, such as collecting data sets from Yelp and processing them in Google Cloud Platform for use in market research.

 

In this scenario, a Yelp data set in JSON is ingested by the Google Cloud Pub/Sub (Publisher and Subscriber) module, while Google Cloud Storage is connected to Cloud Composer. The JSON stream is then published to the Pub/Sub topic. The Cloud Composer and Pub/Sub outputs are processed by Apache Beam and connected to Google Dataflow, before the resulting structured data set is sent to the data warehouse (in this case, Google BigQuery). Finally, BigQuery sends the data to Google Data Studio for visualization.

​

What is business analytics?

​

Business analytics uses information to make decisions and implement changes that enhance financial performance. While business analytics and data analytics both rely on data to benefit business, the two activities are distinct in some ways.

 

Whereas data analytics involves the use of large data sets to discover patterns and produce insights from those sets, business analytics is more focused on the wider implications of data, and the actions taken according to those insights. Broadly speaking, this involves measuring and improving core business functions, such as IT, marketing, or sales. For example, using business analytics can help determine whether a company should provide a new service, or how it should prioritize its existing solutions.

​

Data analytics tools and techniques

 

There are many examples of how data analytics is used to convert raw data into useful insights. But first, you need to ask: which software should you use? And what are the best methods to achieve the results you need? Since data comes from many different sources in many different formats, naturally a multitude of tools have been created to handle all of these. Let’s break down the options further. 

 

Which tool is best for data analytics?

 

The right data analytics tool always depends on the specifications required for the task at hand. Before selecting which software to use, you need to consider which data sources, data types, and data integrations you require. You also need to assess your current data security and regulatory governance, such as data access and sharing permissions.

 

Another consideration is modeling capabilities. For instance, is it best to carry out data modeling manually, or use a ready-made solution? Finally, there’s the question of cost-effectiveness - while there are many free data analytics tools out there, some of these may require more coding skills and manual intervention than a paid solution.

 

Data analytics techniques

 

There are too many data analytics techniques to describe comprehensively, but here are some of the main methods used.

​

Regression analysis

Regression analysis aims to establish correlations between variables. Past data is used to investigate how the value of a dependent variable alters when one or more independent variables change or remain constant. This allows an analyst to identify trends or patterns, which they then typically use to make predictions. 

 

For example, the dependent variable could be new customers while the independent variable could be the number of paid search ads. A positive correlation would imply that increasing the number of ads generates more new customers, while a null correlation would suggest the ads have no effect on the number of new customers, and a negative correlation would imply that reducing the number of ads actually attracts new customers.

 

Factor analysis (or dimension reduction)

Factor analysis aims to determine whether there are any relationships between variables, and therefore reduce a large number of observed variables into a smaller number of unobserved factors. Factor analysis is based on the principle that multiple variables often correlate with each other because they share an underlying construct. Once this is completed for all variables, the resulting smaller number of factors can be more easily used for further data analytics techniques.

 

For example, the amount of different variables constituting a customer profile could range into the hundreds, so it would be easier to reduce these to a more manageable number. Factor analysis can be performed to test which of these variables exhibit covariance and then group them together. 

 

Variables such as ‘household income’ and ‘monthly spend on related services’ may show a strong positive correlation, and can therefore be combined into a single factor labeled ‘purchasing power’. 

 

Cluster analysis

Cluster analysis is another pre-processing step which occurs before other data analytics techniques are applied. It is used to group similar (homogeneous) data points into clusters, while the clusters themselves are dissimilar (heterogeneous). When the number of data records is very large, as is typically the case in the world of big data, clustering overcomes the need to address each individual record.

 

For example, marketers may be unable to tailor their services to each individual customer, but by using cluster analysis, they can combine customers into groups based on similar demographics or buying habits.

 

Cohort analysis

Cohort analysis focuses on segmenting users according to their stage in a defined cycle. So rather than grouping people with similar static characteristics, such as demographics, they are segmented according to their point in a particular lifecycle, which is typically the customer journey. The aim is to determine how changeable the variables are, so that stronger insights can be obtained about customer segments.

 

For example, users may be segmented according to when they first registered on a website, such as the ‘spring’, ‘summer’, ‘fall’, and ‘winter’ segment. Cohort analysis could be used to monitor how each segment responds to different advertising campaigns or seasonal discounts, so that marketers can understand when and how prospects are most likely to convert to customers.

 

Text analysis (or text mining)

As the name implies, text analysis involves evaluating large sets of textual data to understand its intention or emotion, and is usually performed using machine learning and intelligent algorithms. The most common use case is sentiment analysis, which allows marketers to analyze what people are saying about them to monitor brand reputation, and understand how this changes depending on actions or events.

 

For example, text from blogs, reviews, and social media posts could be used to gain insights about users’ preferences. Not only would this inform product development, but it would enable marketers to hone their brand messaging to better match potential customers’ needs.

​

 

What is open-source data analytics?

 

Open-source data analytics simply means the use of open-source tools in data analytics. ‘Open-source’ refers to software which consists of a publicly-available code-base. While it’s possible to use open-source software exclusively, data engineers often mix it with proprietary software to build their solutions.

 

Although open-source components usually come with a lower price tag, they also tend to require more work from data engineering teams to set up and maintain. Since the software also relies on self-hosting the relevant data, the cost of ownership can quickly increase.

 

​

Open-source data analytics tools

There are many developers who have launched and shared their open-source data analytics software online. Here are some of the main tools used today:

 

• Apache Hadoop – The open-source implementation of the MapReduce algorithm created by Google and Yahoo. This is used to efficiently store and process large data sets in the order of gigabytes to petabytes, and is the basis of most data analytics systems.

• Apache Spark – A library used for processing and controlling data flow.

• KNIME – A data pipeline management and visualization tool.

• Redash – Another visualization tool which includes many features, such as alerts, user management, and access control.

• RStudio – This is an integrated development environment for the R statistical programming language, which makes it a very powerful tool for data analytics tasks, such as parsing large data sets via connections and integrations.
   

Data analyst skills

IDC estimated that the global business spend on big data and data analytics increased by 10% between 2020 and 2021 to reach $215 billion. To participate in this market, data analysts need to have a particular skill set.

 

Top technical skills for a data analyst

While data analytics work is varied and requires a vast array of different technical skills, there are some common proficiencies which nearly all data analysts must master.

 

• SQL (Structured Query Language) – Naturally, data analytics involves handling and querying databases, and this is the de facto industry standard database language. 

• Python – While this is a general-purpose programming language, it has a number of specialized AI libraries that make it particularly useful for statistical modeling.

• Data mining – This is used to find patterns, anomalies, and correlations in large data sets, and is a core activity in data analytics.

• ML – AI, and specifically machine learning, is one of the most important tools used to build and adjust predictive models.

• Statistical analysis – Statistical modeling is essential for identifying patterns and relationships between data points, so familiarity with a range of techniques, such as resampling and linear regression, is crucial.

• Microsoft Excel – Although spreadsheets are not best suited to large data sets or complex manipulations, features such as Macros or VBA (Visual Basic for Applications) are widely used for lighter or quicker analytics.

• The R statistical language – This is a powerful statistical programming language, and much faster than Excel when dealing with large data sets and complex workloads.

• Data visualization – While data analysts work primarily with numbers and computer code, it’s also vital to illustrate their findings so these can be easily interpreted by business and leadership teams.

​

Top soft skills for a data analyst

Although many technical skills are essential for data analytics, there are also other important skills which help make a data analyst more effective in their work.

 

• Critical thinking – This is needed to understand and interpret the patterns in data, and the results of data analytics techniques.

• Problem solving – The willingness to find the root cause of a problem is just as important as knowing how to solve it, since the source can often be hidden.

• Research – The information required to make use of data isn’t always available immediately, so some investigation is often needed to interpret raw data or the resulting analytics.

• Attention to detail – Not only do they need to ensure that data is cleaned and prepared correctly, but data analysts must also be able to pick up on signs that indicate hidden insights.

• Presentation skills – Just as data visualization is important to more easily interpret results, data analysts must also be able to communicate their work effectively to others.

​

How to learn data analytics

 

Data analytics skills are in demand, but what is the best way to learn the subject? Let’s take a look at the steps to becoming a data analyst.

​

How do beginners learn data analytics?

Typically, a data analytics career begins with a bachelor’s degree majoring in a subject with statistical or analytical skills, such as computing or mathematics. Once this is earned, the next step is to obtain relevant work experience via internships or industrial placements. 

​

Working on real-world projects enables the development of a portfolio which demonstrates work experience and competency. While on this journey, it’s helpful to develop soft skills too, such as presenting findings. This foundational experience will make it easier to apply for entry-level data analyst positions.

 

How to develop data skills

Here are some tips for developing data skills more quickly:

 

• Certification – Pursue a professional qualification, such as MapR Certified Data Analyst, INFORMS Certified Analytics Professional, or Cloudera Certified Associate Analyst.

• Read and study – Read academic books and papers to improve understanding of data analytics. Some useful books include: Data Just Right: Introduction to Large-Scale Data & Analytics by Michael Manoochehri; Python Data Science Handbook: Essential Tools for Working with Data, by Jake VanderPlas; 1st Edition, Naked Statistics: Stripping the Dread from the Data by Charles Wheelan; and Deep Learning (Adaptive Computation and Machine Learning Series) by Ian Goodfellow, Yoshua Bengio and Aaron Courville.

• Work on projects – Participate in an open-source project, or start a new one, to hone your practical data skills.

Why data analytics is crucial for business 

​​

Data analytics is a cornerstone of any modern business because it provides insights which help improve commercial and operational performance. Without it, data-driven companies would be blind in a world of big data. But conducted well, data analytics can be the difference between failing and flying.

​

​

Switchboard provides a ready-made platform which frees us your data analysts to focus on valuable work which drives the business forward, rather than building and maintaining a software stack. If your company is facing challenges with data engineering, data science, or data analytics, contact us to find out how we could help.