What are the origins of the Polish Graduate Tracking System (ELA)? Were there any assumptions on which to build the system?

The origins of the ELA system date back to 2009, when we had a meeting with professor Mikołaj Jasiński and doctor Marek Styczeń from the University of Warsaw. I then started working at the Educational Research Institute and they came with a question on how we can use the data from the Social Insurance Institution (ZUS) to track graduates. It was more about the work that was then carried out at the University of Warsaw. Prof. Jasinski is the head of the Laboratory of Education Quality Evaluation, and I previously collaborated with dr Marek Styczeń on building a social budget model, where he was involved in forecasting using social insurance data. We started thinking about what we could do, and how we could use the data that is collected by the Social Insurance Institution to track college graduates. This became an inspiration to prepare, at the Educational Research Institute, a project that would eventually be a pilot of methods for tracking graduates. 

At that time, the Ministry of Science and Higher Education introduced changes in the Law on Higher Education, which aimed, among other things, to impose on universities an obligation to track graduates. Because traditional tracking involving surveys or direct contact with graduates after they leave university is very difficult, and the achievable response rate is very low, using social insurance registry data in this situation seems like a good solution. And so the work began. At the first stage, we focused on the data on graduates of the University of Warsaw, which were then verified by the Social Insurance Institution (ZUS). We got aggregate information on how many graduates are in the Social Insurance system, how many have accounts. It turned out that there were a lot of them. We were trying to convince the Ministry of Science and Higher Education and Minister Kudrycka that the implementation of such a system is much better than trying individual, methodologically very diverse ways of searching for information about graduates. At that time the ministry was awarding grants for various kinds of monitoring systems - they were expensive and produced little results. Minister Kudrycka was at first very skeptical about the idea of tracking graduates. She said in the media that it was not a good idea, but, fortunately, a team of young, intelligent, ambitious and world savvy people from the Minister's staff somehow convinced her that it might be worth trying after all. Well, the Minister changed her mind, which is very valuable, and we were actually able to implement the change. 

After the project based on the data of the University of Warsaw graduates had been completed, we deepened our analysis at the Educational Research Institute.  Looking at the process of the collection of information about students and graduates at different universities, mainly through USOS (University Study-Oriented System), we realised that everyone uses it differently - differently in Katowice, differently in Warsaw - which was also very valuable for us, because administrative data are unique in nature. They are the source of information about the population, which is just wonderful for researchers. The quality of the information is only as good as the quality of the data collected in the system, and it is influenced e.g. by whether the person who enters the data does it properly or improperly, whether he/she makes errors, how the data is entered. All of this affects how we can then analyse that data. We did our part from the methodological perspective, the law came into force,  and as a result we managed to launch the first edition of the ELA system for 2014 graduates in 2016.

Could you briefly describe how the system works?

The system, as a rule, monitors the situation of graduates for 5 years after they obtain the diploma, so for us the zero moment is the month in which a person obtains the diploma, and from that moment on we monitor the next 60 months of a person's path in the labor market. We are monitoring additional cohorts - we now have two cohorts with full 5-year monitoring: the graduates of 2014 and 2015. Every year there's another edition, there have been six altogether, and we're adding more information. We monitor a number of things - it's not a single indicator. We already have a whole battery of things that we're looking at. Starting with how many graduates, what percentage of graduates are tracked in the Social Insurance Institution database, then all the job search data - how many months pass after graduation until they find a job. We distinguish between all forms of employment that are covered by social security, whether it is a contract of employment, a contract of mandate or self-employment. We also have data on the employment contracts themselves - it's the kind of contract that is most preferred by everyone, so we separate the relevant data as well. We have information on whether graduates changed jobs, that is, how many employers of different types they had. And we're tracking that, too. We also track earnings, unemployment incidents. If someone is registered in the employment office and such information is also available in the Social Insurance Institution (ZUS), we also record the risk of unemployment - that is, the state of the labour market. The second thing we monitor are, of course, earnings. Also in the months following graduation. And again, all earnings - those associated with employment contracts, on average, for all majors.

We have the entire population for each major. This is the kind of data we show. We show the data in two ways - we have absolute indicators, i.e., what percentage of students have episodes of unemployment, i.e., what are the average or median earnings of graduates in a given field of study; but we also have relative indicators, i.e., the relative unemployment rate, and the relative employment rate, where we compare the risk of unemployment and earnings to the average earnings in the district (poviat) where the graduate lives. Then we take into account the fact that we have very different situations in the labour market - the earnings are different in Warsaw and different in Jędrzejów (a small city in Poland), for example. Similarly, the level of unemployment is completely different in large cities and in rural areas. This kind of information allows us to actually relate to some extent what is happening to graduates to the labour market. This battery of indicators actually looked like this from the very beginning. We are working now on how to divide the graduate population. We have graduates who had work experience before college, during college, or no work experience at all. This is an element that also makes a big difference in the situation of graduates after obtaining their diploma. And we show such data for individual groups as well. Of course, all the majors that we show - here we take into account first-cycle, second-cycle or long-cycle programmes, whether they were conducted as full-time or part-time courses.

There were also new groups this year. For the first time we have relevant information about students, but for now we are showing primarily their pathways while in higher education institutions. It turns out that students very often do not finish the courses they start - they revise their paths by either changing their programmes, dropping out for a while and coming back later, or dropping out altogether. In the future we will also compare the student data with the information about what happens to these students in the labour market, i.e. whether, for example, those who stopped studying did so because they got good jobs, have handsome salaries and, it may seem, that the studies in such a situation were not a priority for them. We also have information about people who have received doctoral degrees, so we also want to look at doctoral students in the context of both their academic career and their situation in the labour market. For example, we have information concerning the earnings of PhD graduates who stayed in academia and those who moved into the non-academic job market. How they differ. We will also supplement this data with information about various kinds of achievements and results of scientific work of PhD graduates - publications, research projects. This is the information we are going to add - the ELA is evolving all the time and we are trying to build this system in such a way that it will provide knowledge to both the Ministry and higher education institutions, as well as graduates and potential candidates - a very large group of people or institutions that may be interested in the situation of graduates.

In what way, in your opinion, is the system innovative? Didn't you have to struggle with technical and bureaucratic issues?

There was initial resistance from Minister Kudrycka. So I can't really say that the ELA was implemented without resistance. The state collects a lot of information, of all kinds, and we must make sure that this information will not be used inappropriately or inadequately. Of course, we also have the data protection act - RODO (GDPR). This is an additional factor. RODO was not yet in place when the ELA was implemented, but now it is an additional factor that makes many institutions very "rigid". When we started working on the first project at the Educational Research Institute and when media information appeared that we wanted to implement such a project, we were contacted by the Panopytkon Foundation, an organization that deals with the verification of the extent to which personal data is adequately protected by public and other institutions. We had a meeting and explained to them what the idea behind the system was, and we actually got some acceptance and support from them.

First of all, the ELA gives you absolutely anonymous data. If the numbers are too small, e.g. very few graduates or very few graduates who chose a specific professional path - we do not publish the data. It is precisely so that specific graduates cannot be identified in any way. Secondly, the whole system of data acquisition results in the fact that the database, which is then processed, i.e. analytically prepared at the National Information Processing Institute, where the system is located, is anonymised. We don't have any personally identifying information such as PESELs, names, surnames, addresses. We know what a person's district (powiat) of residence is, but this is only statistical data at the region or TERYT (National Official Register of Territorial Division) level that we get from the Social Insurance Institution. So here, there is no identifiability. The data is used for report preparation only. Reports in the ELA system are also generated automatically. Each higher education institution gets a dedicated report. The institutions can also generate themselves reports for the majors they offer. Reports are available at the major, institution, and national levels. On the ELA website, we can generate rankings and graphics that show what's happening with graduates - there are multiple ways of getting the information.  The data is also aggregated at the major level and published in the form of a database which is also used by others to conduct analyses, e.g. Professor Rocki from the Warsaw School of Economics publishes a lot of articles analysing the ELA data. The data is also used by the monthly magazine "Perspektywy" to create rankings. So here you can see those applications. The innovation, however, is related to the fact that, first of all, we were able to convince decision makers that you can merge data in a secure way. And this makes sense because we are gaining new knowledge about our graduates that we didn't have. From this perspective, it is a very innovative system. Although in a few countries in Europe, primarily in Sweden, and to some extent in Austria or Hungary, such systems are in place.

I wonder if these were government or grassroots or HEI initiatives?

Arguably, there is always a higher education institutions' interest there. In Sweden, this system is managed by an institution that is equivalent to our PKA (Polish Accreditation Committee). The Swedish system is perhaps more comprehensive, but in Sweden, and more widely in Scandinavia, the use of administrative data is quite different. So that's obviously a different situation, too. In Austria, I feel it's more a result of the universities' interest. In contrast, the Swedes are watching our system very closely. Because innovation also lies in the indicators that we use. Primarily in relative indicators - nobody has done it this way before. For five years, we look at what is happening with, for example, salaries. Obviously we have inflation, all sorts of factors that affect earnings. Comparing earnings in PLN between different cohorts of graduates doesn't quite make sense. What's innovative in our system is that no one has ever thought about it this way - no one has taken into account the uniqueness of local labour markets, which we also take into account. Methodologically, it is probably the best designed system in Europe. Besides, it is highly appreciated by the European Commission, and we know that the Ministry of Science and Higher Education also gets a lot of praise from various institutions that get acquainted with the ELA. We are absolutely proud of how the system is being received, how it has evolved. And, of course, we love to boast about it.

What was the biggest challenge in the work to create the system?

The biggest challenge was, primarily, preparation of legal changes - use of administrative data requires appropriate regulations which describe in great detail what can be obtained and how. That was a real challenge - to get these legal changes initiated and implemented. And here a major role of the Ministry of Science and Higher Education which designed these regulations. There were also some challenges connected specifically with the first rounds, some necessary adjustments - what kind of data was available in the POLON system, what data was available in the Social Insurance Institution (ZUS). We had to negotiate with the ZUS in order to establish the best time when we could obtain the data - they must, first of all, carry out their primary obligations; we established, for example, the month when the ZUS can export data, because in some months, for example March, all pensions and disability allowances are adjusted and then the ZUS servers are working at full steam. So we can get the data only after the first of March when the ZUS servers slow down. Along the way, there have been several amendments of a technical nature so that we could obtain information about the district (powiat) of residence. TERYT (National Official Register of Territorial Division) was just introduced instead of the zip code that was used before as it turned out that the full zip code was missing - we had the first three digits and sometimes it turned out that one code referred to two districts (powiat). This made properly assigning data difficult. We didn't have information about the graduate's possible demise which also made things difficult - if someone is dead, it's hard to be active professionally. So that kind of information had to be provided in subsequent revisions. These were some other annoyances, sort of running-in of the system. Designing new reports, infographics - that's an ever-present part of the job, but I think the hardest part, clearly, was the legislation and just getting the system up and running. It's important to remember that once the system was up and running, we found that not all higher education institutions were entering information concerning their graduates or not all of the graduates. The ELA helped a lot here, because after the first round we learned, for example, that one of the universities declared 15 graduates while it had several hundred, which was actually picked up by some journalists. As a result, higher education institutions also began to enter data in a more timely and complete manner. This was also a challenge - a challenge that led to improved data quality.

Universities have to export data to the POLON system (The Integrated System of Information on Science and Higher Education), if only in order to receive subsidies - so there is a number of incentives to enter correct data. This reporting isn't solely for the purpose of monitoring the situation of graduates. Also, because we have very diverse data in the ELA - on unemployment and employment, and on earnings, it's not like you can always see only the positives. We monitor the data year after year - for higher education institutions it's a source of information on whether their graduates are doing better in the job market or worse. I think this informational content helps universities - they want to know what's going on with their graduates.

Do you feel that since the system went live it has begun to have a visible impact? Does the ELA influence the work of higher education institutions?

It's very hard for me to say whether colleges will change their strategies. On the other hand, thanks to the fact that the ELA data are used in rankings - above all in the "Perspektywy" ranking, which is very popular, many universities boast about the good situation of their graduates. There is certainly a PR piece based on the ELA data. The ELA is also used to support higher education institutions. I'm talking here about state higher schools of vocational education - the Ministry of Science and Higher Education gives the best schools of this type additional funds if their graduates fare well in the labour market. I think these are very direct incentives for improvement and for higher schools of vocational education it is certainly important too.

Do you feel that the interest in the ELA is growing? Is the system monitored in any way?

We certainly monitor the system, although I can't give you any specific indicators at the moment - the ELA works in waves, when new information is published, we have more attention. We also track how the system is received in the media - media reports show that after each launching of another round media interest is quite high. You can see that the information about the ELA is reaching further and further away. I have a feeling that it's not the case, however, that all universities are absolutely aware that the ELA is there, what data is in stored in the system, and, overall, how friendly the system is and how easy it is to generate desired information. We are still trying to spread this knowledge, cooperating with the General Council for Science and Higher Education, or with the Conference of Rectors of Academic Schools in Poland, in order to increase awareness of the ELA. Certainly the visibility of the ELA is considerable, but I think we can still work here, we're trying to work as a team at the National Information Processing Institute, to promote it.

It is worth mentioning that the ELA is an inspiration for others, also in our country - the system for monitoring secondary school graduates that was created at the Educational Research Institute also draws on the experience of the ELA.

What do you think about the system for monitoring the professional and educational pathways of secondary school graduates?

Formally, there was no first full round yet. This project took a little longer to develop, but secondary schools are more complicated. In general, the pilot, which is also done by the Educational Research Institute in a rather difficult arrangement, where it was combined with research conducted by Kantar, a research agency, as part of another project supervised by the Ministry of Education. It was a little complicated, but I think what came up is a very nice solution. More complicated, because one has to take into account not only professional, but educational pathways as well - some graduates after finishing general secondary school or technical school choose higher education institutions. We have far more occupations of all kinds and secondary schools than higher education institutions. The diversity regarding what kind of data and how can be collected, graduate exam results, their subsequent pathways in the labour market and in education makes the system more complex, more extensive. But we are glad that the good experience of the ELA has allowed such a system to be built in Poland. It will be a very important element for the schools, for school governing authorities, and also for the graduates themselves to check the quality of teaching, the outcomes of teaching and what happens to the graduates. I'm keeping my fingers crossed for the ELA's little sister. Once Ms Urszula Martynowicz, former director at the Ministry of Education, called the new system "Elunia" (diminutive of ELA), so we feel a little bit like the ELA is its big sister.