Guest blogger: Using AI for jobseeking

Job Market Finland already included the option to search for jobs matching your profile, and the search was based on factors such as skills and education and the ESCO classifications of competences and occupations. With the updated profile, you can also enter free text on your skills and work preferences, and the search will now also include methods based on natural language processing (NLP). In this blog post, I’ll describe a few AI methods that the current Job Market Finland job search uses.

Relevance model

We found pretty early on that searching the job posting texts directly is somewhat problematic. Job postings may contain a lot of words and phrases that do not actually describe the job in question. For example, a posting by a metal company may include a lengthy description of the company's operations in the metal industry when the company is actually looking for an accountant or cleaner – not a metalworker. Using the posting as-is would overly emphasise the words concerning metal work. To solve this problem, we built a tool for pre-processing job postings, which we call the relevance model. The model seeks to look at a job posting and only extract the words that really describe the job and the right kind of applicant.

In practice, the words in the job posting are first converted into numerical vectors using a FastText template, and then the words are entered into a recurrent neural network (RNN) that calculates the score for each word's relevance. Finally, the relevance model returns the words with the highest score. The number of returned words depends not only on the score, but also on the length of the job posting; the model will return a higher percentage of words from a shorter posting and a lower percentage from a longer posting.

The relevance model has been trained with job postings in Finnish, Swedish and English, so the same model can process job postings regardless of the language(s) they are written in. The training material was created by manually reviewing a few hundred job postings and extracting the most relevant words from them.

Let's look at an example. The words picked by the relevance model are in bold on this job posting:

"We are a reliable employer. We offer jobs in construction, logistics, industry, social and health care as well as the hotel and restaurant and retail sectors. We invest in the wellbeing at work of our employees at all levels of our operations and we uphold a good working atmosphere. We work nationally near you. We are an internationally operating Finnish family business and we employ more than 14,000 people. We are looking for painters for new and renovated sites in Joensuu. We are looking for people who take the results of their work seriously. You work with care are and have solid work experience in the painting field and a versatile knowledge of the latest materials. The work mostly consists of indoor painting jobs. Our requirements are -about 2 years of experience in painting -a work safety certificate -a good attitude, not forgetting a good sense of humour. Please submit your application soon. Include your CV or a detailed listing of your previous work experience and competence. For more information about the posting, please contact Matti Meikäläinen on weekdays from 8 am to 6 pm, tel. 050-12345678. We offer you a happy work atmosphere and a comfortable customer site. Being happy at work and having an encouraging, collaborative atmosphere is important to us. We regularly survey our employees' job satisfaction and develop our operations with the feedback we receive. In addition to a happy work community, we offer opportunities for training and professional development.”

So the relevance model ignored things such as the company description and application instructions, focusing on the most important thing: what kinds of professionals the company is looking for and what the requirements are.

TFIDF search

Once the most essential words have been picked from the job postings, we can start experimenting with different search methods. The TFIDF search is close to traditional text-based searches. It involves picking out a few letters from a word and seeing how often the most relevant ones appear in each job posting. The method works well in situations where the applicant's text and job postings share several terms, such as when “the Laukaa parish is looking for a sexton” and the applicant has “experience working as a sexton in the Petjäjävesi parish”. Another bonus is that the TFIDF search works even when there is no previous record of a profession or term. For example, if a job posting is looking for a “mutary harbuler” and the jobseeker mentions in their profile that they are a “great mutary harbuler”, this search algorithm can link the two together.

Similarly, the downside of the algorithm is that it does not understand context or synonyms, so there is no link between “lawyer” and “attorney”. The TFIDF search is also language-specific, so a Swedish-speaking applicant will not find jobs in their field if the job postings are in Finnish.

Let's test this method on a database with job postings that represent a wide range of professions: “I have a decade of experience in welding stainless steel.” The top result is a posting: "We are looking for a MIG STAINLESS STEEL WELDER for our growing client company in the Oulu area.” That’s understandably a great match.

It's another case entirely when the applicant and posting don't use terms that are exactly the same. For example, the search “I would like to work as a lorry driver.” doesn't match postings for truck drivers.

FastText search

The best feature of FastText models is that they group words related to the same subject by converting words to numeric vectors. With a FastText template trained with job postings, all words in the postings have been divided into 500 clusters, each of which contains a set of words around the same subject. So the words “lawyer”, “attorney”, “law”, “legal” and “judicial” are all in the same cluster. The model also isn't language-specific, so the same cluster includes the Finnish words “laki” and “asianajaja” and the Swedish words “advokat” and “juridikstudent”. The actual search is then based on the words in the job advertisements that match the same clusters as the words in the search text.

The downside of this approach is that it only recognises words previously taught to the model, so the model needs to be retrained from time to time to recognise new professions, for example. In addition, the clustering of words is not perfect in all respects, and individual words can end up in the wrong clusters. And if the search text is very short, the significance of individual words increases, so the FastText search will produce really strange results with some bad luck.

Let's try the same search that couldn't match trucks and lorries: “I would like to work as a lorry driver.” Now the top result for job postings is: “We are looking for cross-country truck drivers for our client company.” It’s a match! So the FastText model understands the connection between trucks and lorries.

How about: "I can make prostheses if you've had to have some teeth removed.” This search returns nothing because the search text does not contain terms that the FastText model knows.

Occupation search

I previously wrote about the Skills Suggester that can use text to suggest occupations, including ones based on the ESCO ontology. Since job advertisements introduced into the system have a specified occupation or several occupations, they can easily be used to create yet one more search algorithm. You enter the search text to the Skills Suggester and then search for job postings with the occupations returned by the Skills Suggester. The search is not language-specific, as the Skills Suggester works in the same way in all supported languages.

In other words, the occupation search does not use the text of the job posting at all. Depending on the situation, the resulting robustness may be an advantage or a disadvantage. The occupation specified in a job posting is not always optimal either, which sometimes significantly weakens the usefulness of this search method.

Let's try the same search as before: "I can make prostheses if you've had to have some teeth removed.” Now, the top of the list has the job posting: “We are looking for a full-time permanent dental technician / dental laboratory technician in Helsinki.”

Combining results

As we can see, different search algorithms have their strengths and weaknesses. None of them is unequivocally better than the others, and it is not easy to reliably assess in advance which algorithm will produce the best recommendations. That's why the actual job search always uses all of them, using suitable weighting coefficients to combine the results into a single list. This way, poor search results from an individual algorithm can’t drag down the overall result too much. On the other hand, if the same job advertisement comes up with all the search methods, it’s likely a pretty good match for the search text.

In reality, the search algorithm is not quite as straightforward. For example, the search includes separate weighting coefficients for job preferences. We also haven't abandoned the original search that's based on occupations and skills; the results presented to the user are compiled from all these different sources.

Our work is not nearly done yet, and the team working on these features still has many good ideas to try out. We do need as much feedback as possible about the functionality of the current version, which is why I am asking for your help. Try logging in to Job Market Finland and fill in your skills and job preferences in particular. Are the job recommendations successful or maybe ridiculous or otherwise irrelevant? Give us feedback to help us develop the service to meet your specific needs!

Gofore is involved in the construction of Job Market Finland, helping to bring employees and employers together.

Heikki Niittylä

Data Scientist

Gofore