Policy Priority Inference: A Computational Method for the Analysis of Sustainable Development

Session 3: Fintech for Sustainable and Inclusive Development

10:05 am – 10:35 am

Omar Guerrero (Alan Turing Institute) – Policy Priority Inference: A Computational Method for the Analysis of Sustainable Development

To watch the video on YouTube, click the ‘YouTube’ button above.


Thankom Arun

And our second keynote lecture is Mr. Dr. Omar Guerrero. Omar Guerrero is head of computational social science research, and leads the Public Policy Program at Alan Turing Institute. He is an economist by training, and has a PhD in computational social science from George Mason University. He enjoys translating computational social science in the useful tools to address the problems of public interest, particularly the issues of sustainable development. More importantly, he was a student of Sheri at Essex, and a former student of Essex University’s a well known Center for Computational finance and economic agents. So with these words, let me welcome Omer for his keynote lecture.

Dr. Omar Guerrero

Thank you very much. Thanks. And thanks, everyone. Good morning. Good afternoon, as well. And of course, thanks, Sheri, and Anindya for organizing this and inviting me. And it’s a pleasure to be here and to see Sheri again, of course. So let me start sharing my screen. Can you give me a thumbs up if you can see perfectly? Well, so yes, I am Omar Guerrero. And I work at the Alan Turing Institute, which is the National Institute for Data Science and Artificial Intelligence of the United Kingdom. And it is a public Institute, so it was created by by the government a few years ago. And Elisa cross disciplinary Institute where the Common Core of topics have to do with data science and AI, of course, but they cut their application cut across various different disciplines. In my case, I work in economics and application of these techniques to public policy. And what I will talk about today is one of those applications, which has to do with sustainable development is a project that has been going on for a couple of years now, in close collaboration with international organizations, such as the United Nations Development Programme, and some of the methods that I will talk about here fall into what it is called today computational social science. Some of these methods have been pushed forward by by some social scientists, such as Sheri for many years. And I think now it’s becoming the time with more computational power and more data to really show what these methods can do. So that’s why we’re working on the Turing to bring this closer to policymakers everyday. So let me start by giving you just a little motivation of what this project is about. And the problem has to do with with governments and public spending. So the main question, the general question is, how does government expenditure affect development? Because I don’t think anyone would deny that development can happen without government expenditure. But then the question is, how effective is government expenditure in certain countries in certain institutional contexts? And more to be more specific, let’s say that our government has a set of goals, and it also has a government programs to achieve those goals. So, these programs can cut across many different issues, from public health to education to preserving the environment. So the question is, how can the government better allocate a given budget to achieve those goals across all these many development dimensions. And therefore, governments usually look at development indicators to evaluate how how good their their programs are performing. So they normally put together a vast set of different indicators. Some of these are famous, for example, from the United Nations Development Goals, where we have around more than 200 indicators to evaluate progress towards these goals.

This means that we are dealing with a multi dimensional problem of development as I said, they can the topics can cover from poverty to poverty alleviation to public health to the environment, and this is very well reflected in the 17 goals, which many of you have heard of, which are the Sustainable Development Goals is the leading international development agenda, established in 2015 by all the member states of the United Nations, these are very broad goals, very broad topics as you can see 17 They are further subdivided in 169 targets with more specific goals in mind and this at the same time are further subdivided in indicators that are used to evaluate the progress of the countries. So, let me give you an idea of this agenda and what is new about it compared to previous agendas like the millennium development goal. So, before there has been a long narrative, a long standing narrative about the need of a multi dimensional approach. So, that comes from before the SDGs. What is new in the SDGs is the acknowledgement that these many dimensions also interact. And we can see this very explicitly in the, in the resolution that the fund that set the the SDGs. In one of the first lines, they acknowledged that this goals are interlinked. So, the whole agenda has an integrated nature. But we can find even deeper narrative about these interdependencies in the context of complexity. So, you can see this is the the portal of the Sustainable Development Goals fund. And you can see several concepts here mentioned, very roughly one is multidimensionality, which I just mentioned, integration holistic approaches. But here’s the new complex, right complexity interlinkages, this is completely new before, for example, with the Millennium Development Goals, we would consider multi dimensional aspects of development. But each topic would be normally treated in isolation in silos. So you would have policy experts working by themselves in in education than another in poverty, another in public transport. But with the SDGs, the idea is that we have to understand how these topics also interact. And therefore, we have to think about complexity and ways to capture these interactions in the quantitative methods that are used to evaluate development. Let me give you an example of this complexity. Suppose we have a central authority, which would be a government in charge of spending public funding across these different topics that I have represented here with some of the SDGs. Suppose this government allocates certain amount to the quality of education, it will, it will, it wants to improve the quality education. And the way it’s going to measure this improvement is through some indicator, for example, the PISA tests. Now, ideally, these funds will translate directly into improvements of indicators. However, anyone who has dealt with public policy knows that this is not always the case. There are many inefficiencies in the process, we’re talking about inefficiencies of lack of capacity, perhaps of the Ministry of Education, or it could be just plain corruption, embezzlement of resources, or public tenders that are probably favoring certain interest groups. So there can be many reasons why we would lose resources in this process. These normally is, governments try to counter this by monitoring. So they will expectations of how much they they want indicators to improve. And then they will evaluate the the ministry or the agent in charge of implementing these public policies. The problem is that when we have interdependencies in account in our social system, evaluation becomes not trivial at all. And here is an example. Suppose that while the government is spending in the quality of education, it also had improvements in public health, and in public transport. So think about children in rural areas that used to work for hours to get to the school, and they didn’t get all their vaccinations. So now with this, with the new improvements, now they get the vaccinations. Now, there are bosses that can take them to the school. So evidently, these kids are going to perform better. And this is going to show in the PISA tests that I mentioned earlier. So these are externalities, positive externalities that are modifying the indicator of the patient. So this improvement in quality of education, had nothing to do without augmenting the budget. And this policy issue, it had to do with the spillover effects. And now think that you have these spillover effects across many different policy issues, many indicators, and that the government needs to come up with ways to allocate the budget efficiently in the most efficient way.

So I hope this gave you a picture of how complexity works in this context. So the next question, obviously, is, are we using the right tools that we have the right tools to address complexity in the context of sustainable development and budget allocation? So let’s see what we have. So, probably the most popular approach across economists is what is known as growth regressions. So you will try to predict GDP growth as a function of different indicators like public health, quality of education, etc. Now, this is quite a rigid paradigm because you normally have one one dependent variable so you only look at GDP So you leave out the multi dimensional approach. And it also has problems in scaling. When you have many, many dimensions to take into account and their interdependencies. These methods just don’t scale well with the kind of data that is available. We also have systems thinking, so systems dynamics, which are popular to look at the world dynamics. But this is all this, these methods are a bit aggregates, so they’re difficult to validate. And they also have problems scaling up when we augment the space of dimensions, because you have to come up with stories of how different indicators interact. Well, ideally, we would want the data to tell us that rather to assume and impose certain structure in the system. There is this also flavor of complexity that talks about networks that we should think about network. So take data on indicators, construct networks based on indicators, and then make claims of how an indicator impacts the other. Well, this is informative about structure of data. But it’s not really useful for policymaking. Because these networks are built with aggregate indicators, it’s very difficult to tell anything causal about these relationships that one is, is putting into these networks. And finally, more recently, we have machine learning, which is just make predictions based on any data set that you have, which is very useful, exactly to make predictions, but is very difficult to make causal statements about effects or impacts. So my personal take here, and I think Sheri will agree with this, since this method that she has been promoting over the years, is that we need agent computing, also known as agent based modeling, or multi agent systems to actually model the policymaking process that takes us from an agent spending resources in a policy issue at the micro level, to observing the aggregate outcomes of such expenditure in the indicators, so we want to simulate agents that are implementing these policies. And as a consequence, moving the indicators.

Anindya S. Chakrabarti


Dr. Omar Guerrero


Anindya S. Chakrabarti

yes. Here. Yeah. I can, you know, I see where you’re coming from and its a actually very interesting insight. But what I was thinking is that the standard prescription is to go for RCTs. Or you can use difference in difference estimators, you know, to do causal econometrics, and RCT has been also given Nobel Prize for basically these kind of estimates and policy prescriptions.

Dr. Omar Guerrero

Yeah, of course, what RCT are not scalable in reality, right. So I presented you with 17 goals. So I don’t know any RCT that can measure 17, or 100, different dimensions, RCT are very good for a very specific context of a very specific population in which you can control very well and randomize the group and then control for the treatment, right. So they’re only useful to tell you about that specific context. When you want to talk about government spending across different topics, RCT are not really useful. And, and a similar criticism goes to the difference in difference, because in order to justify your instruments and make causal claims, you really need to narrow down a lot your context and the specific situation that you’re observing in the data. So I hope that more or less puts in context the discussion

Anindya S. Chakrabarti


Dr. Omar Guerrero

so this, this method, as I mentioned, they have been promoted by several academics across the years, specifically aging computing. And so they are not new. But in the context of sustainable development, they have not been very popular. So they’re popular in in places like in epidemiology, in conflict analysis, in sociology, but not so much in sustainability. And actually, in 2016, right after the SDGs were established. Allen and co authors did a survey across government documents, to see how popular were different computational models, and that this is the outcome. So you have more traditional rhetorical macroeconomic models for use in the context of sustainable development, computation, computable, general equilibrium models, systems dynamics, and here are agent based models, one person, so it’s really not popular and what they say is that it is very promising indeed, but they can get complex and so far, according to their view, these could be the benefits were not that clear. So they had limited like practical applications. So clearly, there were pragmatic issues that were limiting the adoption of agent base. So the project that I will talk to you about now are trying to break those empirical limits and bring these methods to be useful in the context of public spending across complex multi dimensional space. So the project is called policy priority inference, like the title of the talk. And let me give you some context. Now on the empirical side. And I will give you a context with one of the first projects that we conducted in collaboration with the United Nations Development Program. This project was focused on Mexico. So in Mexico, every time there is a new government taking office, as it happened in 2018, the government has to publish a national development plan, governments in Mexico last six years, so that government plan should specify a set of objectives that they would want to achieve across a multi dimensional policy setting, and specific values that they want to achieve for indicators classified into the subject is, and they want to achieve these goals in six years. So this is a picture of such document. It was published in 2019, which is when the government takes office, it considers 200 objectives. Now, specifying indicators for the for objectives is still a work in progress for most countries in the world. So out of those 200 objectives, actually document has 60 indicators, not 200 indicators. And something cool. That has been an innovation of the government in Mexico has that they have mapped all these indicators into the SDGs. So we can actually how a narrative in the context of sustainable development. These are some examples. Obviously, they’re in Spanish, and I apologize, some of the things I will show are in Spanish, since this is work whether we have conducted with local authorities, but I will do my best to translate some things here. Although I think I think some will be a bit self evident. These boxes are extracts from these documents. And I just want to show you some of them to show to exemplify how specific they can be. This is an objective with an indicator which has to do with carbon emissions. It describes the indicator, it describes a baseline that indicator has in this case it had in 2016. And then it establishes the goal that it wants to reach the government wants to reach in 2024 by the end of the term. So you can see that from the baseline to the goal, the aim is to minimize nearly by half the amount of carbon emissions in the country. And just like this objective, we have one for health services for energy, autonomy or independence, and also for informal labor. And even if you look at this type of documents, you can see already how ambitious is a government or how realistic it is or how unfeasible some goals might be an example here. The last one is the the one of informality, you can see that in Mexico in 2019 56, almost 57% of the population of the labor force was informal. And the goal in six years was to reduce it only by two points to 3.8%. So it’s a very modest goal. And this comes from the fact that Mexico knows that it is difficult to actually to hit these indicators, the government programs historically have not shown to be very effective.

So I would have this type of documents. And this is an example for Mexico. But we have seen these documents for many countries in Latin America, probably in India that are similar countries, probably at the local level, as well as state levels. This is a common practice. And sometimes even if they are not official documents explicit in one way or another or another, governments tend to have goals in mind, even if it’s an in a more implicit way. So some of the relevant questions have to do with how long would it take to reach the goals that governments have set? How realistic are those goals? Would we reach them actual in six years or not? Is a national budget that the government has approved or the Congress has approved? Is it coherent with these goals? In the sense that is this the best budget we can configure to reach those goals? Or we could actually save time in reaching those goals if we shift around resources? And we optimize them this budget? And can we then identify bottlenecks and accelerators. So topics that are very promising, and that would catalyze development and bottlenecks would be topics in which no matter how much money you pouring, you’re not going to improve the topic. So they need probably more fundamental or structural changes. And in that case, when you identify a bottleneck, that’s when you call precisely the experts that an India was mentioning so you bring an expert with an RCT now to work on that very specific topic and tell you what is wrong. What is the bottleneck? But from a macro perspective, first you need to find them. And we believe that this method helps us to do that. You can think of the method as a computational simulation. So that takes inputs. So as a policymaker, you bring your development indicators, your goals, you also construct this network of interdependencies. This could be through quantitative methods like Bayesian methods, or you could just put experts in a table to determine what is the network. And if you have expenditure data on how much is spent, in the different programs that impact indicators, you can bring it as well. And with the simulation framework, you can address all these topics related to the questions that I just mentioned. This is a general structure of the way the model works at the bottom, you have a simulated central authority that takes this body of data that you give it an allocates it to the different policy issues, the different policy issues are managed by public servants. So artificial agents that represent public servants, or ministries or entities that are in charge of implementing the public policy, these public servants will use some of those resources towards the public policy. Some of the others are inefficiencies. That’s when the inefficiency social norm emerges. We also have mechanisms of public government governance to try to minimize that level of inefficiency. But at the macro level, once these resources are translated into public policies, you will have movements in the indicators and these movements will also produce spillover effects. So you will, you will have now co movement with the indicators and then the spirits inputs both at the macro meso and micro level that we can analyze through different techniques. So I will show you some of the things that we have done with different projects just to be more pragmatic about the framework rather than diving into theory. So going back to the project with Mexico, this is an example of some of the results that we showed, we wanted to know how feasible were the goals that they establish in in the document that I shown you? So what we did was to estimate how long would it take to get to those goals once we had calibrated the model. So we deliver the model with historical data and indicators, and then we simulate forward in time. And what we find is that here you have the indicators, colored according to the SDGs, there are more than 100 that we gather in the project, the stars mean that there are actually specific government programs that are designed to impact those indicators, if it is a circle is what we call a collateral indicator, which means that there are not really specific programs in place. So these indicators move because other factors like spill overs or maybe the private sector, etc. Or maybe the indicator is to aggregate to claim that there is a specific government program. What what you see here is that on average, it will take the government more than 10 years, the reach the goals that they established, so not six years, so they’re not feasible in six years. But they are neither crazy. So you know, if the government would be reelected, or at least the party because in Mexico re election is not allowed, but part of the party could be reelected, then they could this could be a feasible plan.

So it’s a multi term, objective. And obviously, there is some variation, there are indicators that will be very easy to achieve others that will be very hard. So this is very useful to governments right to assess whether their goals are realistic or not, and then give them excellent information, some priors to inform their proposal of the budget that they will come forward to with them with the Congress. Another exercise that we do a lot is what these that has to do with policy coherence and quantifying this concept. So as I said, coherence has to do with is this budget, the best I can have to achieve the goals that I have? Well, what we do here is to try to find whether in the proposed budget that was approved in 2019, which is also classified into the SDGs. In that budget, the government is overspending or underspending in certain topics. So each color would be the aggregated budget at the level of each SDG. And we estimate if the shadow is above the the line, it means that the government is proposing an overall expenditure. So it’s much more than what they would need to achieve the goal. That is the case for energy. And actually, it’s funny because after we publish this, all this came out. All these criticisms to the government came out in the media talking about that their strategic has, has been too centered on energy and carbon emission. So there’s a very big push for building refineries and concentrate on petrol, which is a historical debate in Mexico. There are other topics like, you know, climate action in which there are clearly under expenditure. So it’s also very helpful to the government, we can build precise indices to assess the level of coherence of this entire budget. We have done also work at the sub national level, these are the 32 states of Mexico, the color denotes their level of development. But the quick story here is that Mexico has a very peculiar federal system in which the states have no capacity to collect taxes, everything is collected by the Federation or almost everything. But then the Federation distributes, again, everything to the states and the states have full autonomy to spend that money. How is it distributed back to the States? Well, there is a law that is approved every year by the Congress that approves a formula, a mathematical formula to distribute these back to the States, and that formula themes to compensate for things like population size, level of development, etc. And the question is, given a set of goals as a federation, is that formula, the best one is that redistribution, the best one. And we can do here a similar exercise as the one I showed you. As I said, if there are states in which the these redistribution is excessive, or if they could do with, they could do with less resources. Here’s another piece of work that we did with Uruguay, what we wanted to do is to assess the the topic of acceleration. So using some more sophisticated optimization methods, we tried to find budget allocations in which we could identify those indicators that respond better to public expenditure. So those that are sensitive, and those are the ones we label accelerators. So what you hear when you see here is a number of years that would be saved against the benchmark to reach the SDGs. And the red stars are those that we find the most sensitive, as you can see, they save a lot under an optimal budget. We also have work more work at the sub national level. This is a report that we did with the UNDP for the capital of Peru, Lima, in which we try to identify now not accelerators, but bottlenecks. So those are those indicators that are very insensitive to changes in the budget. And not only they are insensitive. So hearing sensitivity would be in the in the vertical axis, these are number of months saved. So you have less than 60 months saved to reach the SDGs. And we consider them insensitive. Not only they are insensitive, but their level in general is also bad. So they perform bad and they are insensitive. So these are red flags. And then this is when the government should not bring the experts to run their cities in very specific topics. To learn more about what is going wrong.

Then, now we have the also the possibility to assess different budget configurations. These are just examples of what we do to propose different arrangements of the budget, we’re flexible to incorporate rigidity, fiscal GDP, we also have translated this into a flow chart through which the governments can look at all these results that are provided, and then make decisions on whether Okay, here is a bottleneck, maybe I should revise the design of my program. So I would bring these experts that do our cities and difference in difference. Or maybe the indicators are doing fine. And they would respond if I just change the budget. So I don’t need to bring the experts right now. Or just the indicators are great, and they will achieve their goals. So this has been super useful for policymakers. We also have more academic Global Studies, this study in which we evaluate the feasibility to reach the SDGs across the world. Then we we can classify countries according to the bottlenecks. On the left, we have the level of of bottlenecks across different countries. And we can rearrange the same data across SDGs. And presented on the right hand side. I have to go a bit quick now which I’m running out of time. We also are working on things like how do we measure the impact of international aid this is very difficult to do and we just cannot do it at this scale across countries in a multi dimensional setting. So with this methodology, we have been able to do that and provide some insights on what has been the impact in the last 20 years of international flows. And to close I want to just give you these references if you’re interested in some of these work that are the policy reports some of them are in Spanish there are other sin English. These OECD book has a chapter on PPI this was not written by us was written by the UNDP people. So if you want to have a Third Party view on the mythology from policymakers, I invite you to read this book. And also if you go to my website, you will see all the academic papers and this would be I would say the the most holistic one that you could take a look at. I also want to thank my my co author and development PPI on Sela Preston Yetta is a professor in Mexico economics, all the partners that have helped and supported this initiative. And before closing, I want to do some shameless advertising. So I invite you to look at the Alan Turing Institute, specifically the Public Policy Program, which is where my group is part of. And in particular, my group is on modeling for policies, the theme of modeling for policy, where this is one of the products that we have AI for development. But we’re also working on things like labor dynamics, informality, production, networks, and shocks, technological change, inequality, housing, markets, etc. And I am growing my group. So if you know someone who is interested in these methods, and in research, let them know I am hiring. I’m opening positions to hire postdocs on computational social science. And that’s everything from my side, I want to thank you again, the organizers, and pancam for moderating this, and I’m very happy to to answer any questions that you may have.

Thankom Arun

Thank you. Thank you. Thank you, Omar, for illuminating lecture we are so lucky. We have two fantastic lectures today, and priortising issues for maximizing impact. It’s an enormous challenge for all the emerging economies and all the governments in that way. And this is a very pioneering policy tool to understand the progress towards achieving a particular goal. So what I see here is, is a more of a learning element there. The other one is once you implement a program, then it’s very difficult to go back. And it gives quite a lot of windows in beginning to go and rectify. And also it’s a combination of various things that can be based on social models there. But you can see the economic theory, the behavioral economics, network science, and all these are actually very much collaborated in this kind of approach. My only concern is probably we can discuss it later on. My only concern is to what extent the Political Economy elements will allow a government to go back from what they have promised in the beginning of the program, but it’s a fascinating Thank you