What a data scientist wants HR to know about AI skills tools

While 98% of businesses plan to move towards becoming skills-based organizations, there’s still a lot of uncertainty about which tools and frameworks are needed to bring these visions to life. Since both employees’ capabilities and business priorities continue to evolve, organizations need a way to dynamically track the supply and demand of skills—which is where AI-powered skills intelligence tools come into play.

Until recently, organizations lacked the digital infrastructure required to embrace skills-based strategies that would enable leaders to dynamically reallocate talent based on capabilities and capacity. Thanks to the rise of skills intelligence tools like Gloat’s Skills Foundation, that’s about to change.

To gain a better understanding of how these tools work and their potential to transform HR strategy, we sat down with one of Gloat’s own experts, Gabriella Vagner. As one of the data scientists who created Skills Foundation, she’s got plenty of actionable advice for using AI-powered skills tools wisely as well as key considerations that HR leaders must keep in mind.

Give us a little background about you and your role. What do you do in a typical day and how long have you been at Gloat?

GV: I’ve been a data scientist at Gloat for over two years. I got my Masters in Mathematics and am very data and logic-oriented. What excited me about Gloat was the opportunity to work with textual data to create something that impacts people’s lives.

Part of what I love about data science is that no two days are the same. I might be training a new model, cleaning the data, or thinking about possible pitfalls and edge cases that might pose a challenge in the future. It’s a lot of working with data but also with people. There are some days when I’m in front of my computer, reading research papers and writing code, and others when I’m in meetings all day to understand the work that needs to be done next.

It seems like “AI” is everywhere today. As a non-data scientist, how should we be looking at different AI offerings to assess which ones are the real deal?

GV: My top tip is to define your needs first. Think about what you want to achieve and explore various AI systems and use cases to see what might align with that. You have to know about the accuracy, the stability, and the quality of the results. 

I’d also encourage you to think about your expectations for edge cases or scenarios that fall outside of the norm. Do you care if the model you’re using misses some results but you can trust the results it generates? Or is it more important to see as many results as possible, even if some of them are false positives? The answer will vary depending on how you’re using AI. 

Right now, free text generation like ChatGPT is what’s getting everyone’s attention. But we need to be careful with it because the way these models work is they’re meant to mimic human speech. So you have to be extra careful and think about what’s being written and its potential pitfalls. These models are built on the internet and not everything on the internet is true—so there’s a chance that their outputs are inaccurate or potentially biased in the same ways that our society is. That’s why it’s so important to constantly evaluate what AI generates and not just use it as is.

What were some of the goals you and your team established when you began to create Skills Foundation? How did those objectives evolve as you progressed?

GV: To understand what we’re trying to achieve with Skills Foundation, it’s important to know a little bit about how Gloat works with skills. Skills are strings (sequences of data points) that we are embedding into vector space (representations of data across multiple channels). Since there are multiple ways to say the same thing, we wanted to create a way to understand what a skill means, regardless of how it might be phrased in a resume or CV.

For example, one person might say they know Python development while another could write they’re competent at Python as a programming language. Even though the words differ, the meaning is the same. So we wanted to give companies a way to comprehensively understand the skills their workforce has, no matter how they’re written. 

I’ve been working on the ontology (internal database for sourcing and identification) that Skills Foundation is built on since I came to Gloat. I developed a model called the harmonization model that can evaluate the similarity between two strings in the context of skills and phrasing. With my model, we can understand when two phrases are describing the same skill and when they aren’t. Our ontology is the heart of Gloat, it’s what holds everything together. So we knew we had to get it just right, which is what we’ve achieved with Skills Foundation.

Can you break down what large language models (LLMs) are and what’s significant about them?

GV: On a high level, the models take an input, vectorize it, put it through several mathematical operations, and generate an output like “yes” or “no” or on a numerical scale. The hard part is vectorizing the input and performing calculations.

If you stack vectors together, you get a matrix which can be used to perform calculations. The larger the matrix is, the more complicated it’s going to be. Large language models are the equivalent of a huge brainstorm because they’re trained on all of the data available on the internet. It wasn’t possible a few years ago but thanks to technological advancements, it’s getting easier and easier to train those models and even imitate people in some cases—which is what we’re seeing with natural language processing.

What are some of the challenges or limitations of using LLMs in skills technology?

GV: Everyone needs to remember that they’re trained on human data, not the truth, so bias is very important to think about. For example, the model that was trained for ChatGPT was trained on CVs from the internet, so it probably saw a lot of people say they have people skills, even if they don’t. So, if you don’t write that you have people skills, it might deduct points for it. 

Similarly, data analysis is a skill that the model uses. As a result, no matter what role you’re applying for, ChatGPT is likely to tell you to improve your data analytics skills. Whereas in the past, data analysis was a skill that was normally reserved for more technical roles such as financial analysts or bankers.

How are LLMs used in Skills Foundation?

 GV: Unlike other solutions, Skills Foundation doesn’t rely solely on the outputs from LLMs. It’s one component, but it goes through a pipeline of other in-house models and a lot of internal logic that helps us create our taxonomy. 

LLMs are just another tool in our toolbox. We use them, but with a grain of salt. We don’t output raw LLM data to our clients. Instead, it goes through a process of cleanup, approval, and evaluation. LLMs are trained on so much data and have a lot of knowledge about the workforce so we do use it but we never over-rely on it.

Tell us something most Gloaters don’t even know about Skills Foundation

GV: I’m not sure everyone understands how much potential there is. We’ve created a place where people can manage all of their workforce skills in a very accurate way. If you want to learn how many people know how to code, you no longer have to separate it into categories like Python and Java; instead, Skills Foundation gives you all of this information in one place—and that’s just one feature. 

There are other important features, including providing clear visibility into people’s skills which are accurate and not just estimations. You can then see how these skills compare to the project openings at your organization. Skills Foundation makes sure the entirety of employees’ skills are represented, in turn improving workforce planning because executives know what types of skills they have and which type of capabilities they need. It gives everyone more visibility and better access to opportunities.

To learn more about how Skills Foundation works and the role it plays in bringing skills strategies to life, check out The Ultimate Guide to the Skills-Based Organization.