Data Analysis Skills Shortage: Challenges and Solutions

The demand for professionals skilled in data analysis is skyrocketing. The UK government report “Quantifying the UK Data Skills Gap”, found a shortfall of 178,000 to 234,000 individuals with data competency in the UK. It also found that nearly half (48 per cent) of businesses are recruiting for data roles, with 46 per cent struggling with recruitment. In October alone there were 4853 data analyst roles advertised across the UK and a quick and dirty search for skills gaps within data analytics in 2023 shows industry reports from marketing to HR claiming the biggest skills gaps in the respective functions is data analytics. Marketing week report, for example, that amongst its readership (presumably a mixture of agency and client-side) the biggest gap by far is data analysis. A good time to be a data analyst then!

The Skills Shortage Dilemma

This discrepancy between the supply and demand for data analysis skills has created a significant challenge for organisations. It is, as I crassly stated earlier, a good time to be a data analyst but that doesn’t help organisations to cover the gaps they are experiencing right now. They continue to struggle to find the right talent to harness the power of data.

Causes of Data Analysis Skills Shortages

What are the well documented factors contributing to such a gap? For starters data analytics provides such value to businesses that what once might have been a niche role within a few companies, now has spread across most functions within businesses, so the available resource is spread even thinner. Industry-wise it has been reported that 60% of those professionals in data analytics work within technology and IT, followed by industry at 17%, finance 15% and 7% in healthcare.

For some large enterprise level businesses each function would ideally have its own analyst – so HR would have a professional that understands the HR and people function and the data that helps a business grow, pivot, get the best out of the team. Whereas the finance team would have someone who works with the figures and reports to the commercial team/SLT to help steer business decisions.

According to a 2021 Ipsos Mori report the number of people leaving university as qualified data professionals entering into the workforce will not exceed 10k per year - a complete mismatch to the estimated 178,000 jobs requiring data skills.

So what can be done to address this. Against this backdrop we are advising clients to really drill down into their skills requirements and check that historical skills are still relevant for roles. One area that some businesses are missing a piece on is ensuring that what they offer is what the market is looking for. When we look at the current demands, many candidates are looking for career progression, they are looking to be upskilled, they are looking for clear pathways and want to know that within their role they are able to keep up which the rapidly changing technology and skillsets. With 80% of businesses reporting that they are unable to find the skill sets they need to execute on business goals and if you offer these kinds of benefits within your organisation, make sure you are publicising them well.

Our State of Data report indicated the areas where companies could focus their attention to attract top talent. We asked respondents to rank their primary reasons for departing a company. Leading the list was poor management, cited by 27% of respondents. These figures closely align with a recent CIPD report, in which 21% of the surveyed population (weighted to represent the UK population) identified poor management as a potential reason for leaving their jobs. It is crucial to conduct employee satisfaction surveys to detect issues early and mitigate attrition levels. The next significant factor driving employees to consider leaving their positions revolved around low salaries and unchallenging work. The same CIPD survey indicated that a higher proportion of the population is struggling to meet their financial commitments compared to 2022. To address the risk of employee turnover due to low pay or perceived low pay, it is advisable to ensure competitiveness through salary benchmarking and benefits assessments.

So if poor management is a driver for respondents leaving a job, what are the motivators to join a business? The data reveals that a significant motivator among individuals is their capacity to shape business strategy. In 2022, more women than men expressed motivation in this regard, and this trend continues into 2023, with 2% more women than men desiring the ability to influence business strategy. This motivational factor aligns with the concept of meaningful work and the well-established autonomy, mastery, and purpose theory. This theory suggests that once our basic needs are met, our motivation shifts away from monetary rewards and towards autonomy, continuous learning, and a sense of purpose in our work.
 

To ensure you remain competitive and to enhance the effectiveness of your recruitment outreach efforts, consider highlighting specific areas to attract a diverse pool of applicants. These could encompass:

• workplace policies (such as flexible hours, hybrid, remote, or office options, diversity, parental leave, menopause)

• corporate social responsibility initiatives related to charities or environmental targets

• opportunities for career advancement

• reporting structures

• organisational culture

• salary information. *It's worth noting that candidates are more inclined to apply for roles where salary details are provided, with statistics suggesting a significant impact on application rates.

Beyond the scope of the data survey, drawing from extensive candidate interviews within the data community, it's evident that data professionals are drawn to employers who take their data functions and strategies seriously. This sentiment is echoed in the "other" comments collected during the survey, with respondents expressing their desire to work for organisations where "data is taken seriously."
 

Beyond securing the existing talent and plugging any leaks that contribute to attrition levels, are there any other tactical hiring strategies to consider?

With the rise of low code no code technology, many more people are able to get involved with the data function. Albeit with less expertise but as an interim looking at introducing this technology and upskilling from within could be an option for companies struggling fill their roles.

Addressing the data analysis skills shortage requires a multi-pronged approach:

1. Promoting Education and Training

Encouraging the development of data analysis programs in educational institutions can help bridge the skills gap. Scholarships and incentives can attract more students to pursue data analysis as a career. Many companies get involved in young talent initiatives and sponsor events like hackathons to encourage and attract new talent.

2. Leveraging Data Analysis Tools

As mentioned earlier, the advancement of user-friendly data analysis tools allows individuals with limited technical expertise to perform data analysis. This widens the talent pool.

3. Encouraging Cross-Disciplinary Collaboration

Fostering collaboration between data analysts and professionals from other fields can help address the shortage by combining domain expertise with data analysis skills.

The shortage of data analysis skills is a significant challenge for businesses in the data-driven world. However, by promoting education and training, leveraging data analysis tools, encouraging cross-disciplinary collaboration, and implementing hiring and upskilling programs, businesses can address this issue and unlock the full potential of data to drive their objectives forward.