Matching engine, skills clustering, and trajectory models

Why keyword matching fails at scale

The first generation of HR matching was keyword overlap. Take the words in a job description, compare them to the words in a resume, count the matches. This approach has three fatal problems.

First, it misses semantic equivalence. “People management” and “team leadership” mean the same thing, but keyword matching treats them as unrelated. Second, it cannot handle adjacency. A software engineer with Python, data pipeline experience, and statistics knowledge is a strong candidate for a data engineering role, but keyword matching only sees partial overlap. Third, it is backward-looking. It scores based on what someone has done, not what they could do next.

These failures compound at enterprise scale. When you have 50,000 employees and 2,000 open roles, keyword matching produces noise. Managers stop trusting the recommendations. Employees stop checking. The system falls into the adoption trap described in Article 2.01.

Multi-dimensional matching architecture

Modern matching engines score across multiple independent dimensions and produce a composite relevance score. Each dimension captures something different about fit.

Each dimension produces a normalized score between 0 and 1. The composite score is a weighted combination, where weights are configurable per use case. A succession planning scenario might weight trajectory alignment and growth potential heavily. A gig marketplace scenario might weight skills match and availability.

The critical architectural choice is embedding-based similarity rather than rule-based filtering. Skills, roles, and people are mapped into a shared embedding space where proximity indicates relevance. This approach handles semantic equivalence natively (“people management” and “team leadership” land near each other) and surfaces non-obvious adjacencies that rule-based systems miss entirely.

Inside the matching pipeline

A production matching engine runs through several stages for each query.

Stage 1: Candidate retrieval. The system uses approximate nearest neighbor (ANN) search in the embedding space to retrieve the top N candidates. This is a speed optimization. Scoring every employee against every role is computationally expensive; ANN search narrows the field to a manageable set in milliseconds.

Stage 2: Multi-dimensional scoring. Each candidate-role pair is scored across every dimension. Skills match uses ontology-aware comparison (understanding that “Python” is a subset of “programming languages” and adjacent to “data engineering”). Experience fit uses contextual embeddings of role descriptions. Trajectory alignment checks the proposed move against the trajectory model.

Stage 3: Business rule application. Configurable rules filter or re-rank results. Examples include minimum tenure requirements, geographic constraints, diversity considerations, and manager approval gates. These rules are separate from the ML scoring, so they can be modified without retraining models.

Stage 4: Explanation generation. For every recommended match, the system produces a human-readable explanation. “Recommended because: 85% skills overlap, strong trajectory alignment with stated interest in product management, and 2 of 3 gap skills available through existing learning programs.” Explainability is not optional in HR context.

Skills clustering: seeing the forest

Individual skills are useful but limited. Knowing that an employee has “SQL” and “Tableau” and “statistical analysis” and “A/B testing” tells you more when you recognize that these cluster into a “data analytics” capability domain.

Skills clustering algorithms (typically hierarchical clustering or community detection on a skills co-occurrence graph) identify groups of skills that appear together frequently in the labor market. These clusters serve multiple purposes:

• They reveal capability domains that are invisible in flat skill lists
• They identify transferable skill sets across seemingly unrelated roles
• They highlight emerging skill clusters before they appear in formal job taxonomies
• They power gap analysis at the organizational level (“we have strong individual data skills but weak data analytics clusters”)

The clustering is dynamic. As market data updates, clusters evolve. “Prompt engineering” did not exist as a meaningful cluster three years ago. Today it clusters with “AI application development,” “LLM fine-tuning,” and “evaluation methodology.” A static taxonomy would miss this entirely.

How clusters are built

The process starts with a co-occurrence matrix derived from millions of profiles and job postings. Skills that appear together frequently in the same profiles or the same job requirements get high co-occurrence scores. This matrix becomes a graph, where skills are nodes and co-occurrence scores are edge weights.

Community detection algorithms identify densely connected subgraphs within the larger skills graph. The result is a hierarchy: broad domains (e.g., “Software Engineering”) containing sub-clusters (e.g., “Backend Development,” “DevOps,” “Frontend Engineering”) containing individual skills.

Trajectory models: predicting the next move

Trajectory models answer a forward-looking question: given this person's current skills, experience, and context, what are the most likely and most valuable next career moves?

The training data comes from historical career transitions across a large population. If 10,000 people with a similar skill profile moved from Role A to Role B, and 80% of those transitions were successful (measured by retention, performance, and promotion velocity), then that transition path has a high trajectory score.

Trajectory models are not deterministic. They produce a probability distribution over possible next moves, accounting for:

• Historical transition patterns for similar profiles
• The employee's stated career interests and aspirations
• Organizational demand signals (which roles need to be filled)
• Skill gap feasibility (how realistic is it to close the gaps for each path)
• Market trends (which roles are growing, which are contracting)

The practical output is a set of recommended paths, each with a probability score, a gap analysis, and a development plan. “Path to Product Manager: 72% trajectory score. Gap skills: product strategy, roadmap planning, stakeholder management. Estimated development time: 6-9 months. Available learning resources: 4 internal courses, 2 mentors identified.”

Market intelligence layer

Matching, clustering, and trajectory models all improve dramatically when they incorporate external market data. Internal data tells you what your workforce looks like today. Market data tells you what the world demands tomorrow.

The market intelligence layer ingests these signals and feeds them into the matching and trajectory engines. A role that requires “AI safety” skills gets flagged as hard-to-fill based on market scarcity data, which changes the matching weights (the system looks harder for adjacent candidates who could upskill). A trajectory model incorporates market demand to avoid recommending paths toward roles that are contracting.

Putting it together: a real scenario

A VP of Engineering asks: “I need to staff a new AI platform team. Who in the organization could transition into these roles within 6 months?”

The matching engine retrieves candidates based on embedding similarity to the target role profiles. Skills clustering identifies employees who have strong “ML engineering” and “platform engineering” clusters, even if they have never held an “AI platform” title. The trajectory model scores each candidate on transition feasibility, factoring in their current skill gaps, learning velocity, and career interests. Market intelligence flags that “MLOps” skills are scarce externally, making internal development the faster path.

The result: a ranked list of 15 internal candidates with match scores, gap analyses, development timelines, and recommended learning paths. The VP reviews in 10 minutes what would have taken a talent acquisition team two weeks to research manually.