THE WORLD BANK OPERATIONS EVALUATION DEPARTMENT OED Expert Knowledge Review Primary and Secondary Education Maureen Woodhall Director-General, Operations Evaluation: Gregory K. Ingram Director: Ajay Chhibber Manager: Victoria Elliott Task Manager: Catherine Gwin, Lead Evaluation Officer, OEDCM 2003 This paper is available upon request from OED. The World Bank Washington, D.C. 2 ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) e-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/oed 1 September, 2002 OED EXPERT KNOWLEDGE REVIEW PRIMARY AND SECONDARY EDUCATION This report is written in the context of a wider OED evaluation on how effectively the Bank is carrying out its commitment to becoming a global knowledge institution, and sharing such knowledge with clients and other development partners. The sub-sectors that were selected for evaluation in education are primary and secondary education, the areas that represent the mainstream of Bank lending. (See the terms of reference of this report in Annex 3). I. Approach The evaluation proceeded in several steps: 1. As a first step, the following seven countries were selected by OED, on the grounds that they represent a range of borrowers in several regions: Bangladesh, Brazil, China, Malawi, Pakistan, Poland and Tanzania. 2. The second step was to select individual primary or secondary education projects in the above countries, after 1996 (i.e., after the Bank made its global knowledge commitment). Given such restrictions, the six projects listed below were automatically "selected", as there were very few alternative candidates in the education sector. 3. In parallel with the above, several pieces of major analytical research and economic and sector work (ESW) were identified, that have generated or disseminated knowledge relevant to primary and secondary education operations. 4. In addition to research and ESW, the Bank has sponsored or participated in major events pertaining to primary and secondary education, and these are also included in the review. 5. Finally, and given the increasing use of electronic databases, the relevant Bank e-sites were also the subject of this review. Annex 2 gives a complete list of the material that was initially identified for review. During the review process a few minor adjustments were made, as reported below. 2 II. Rating To facilitate comparison across sectors, this report uses the item summary template and evaluation rating scale supplied by OED; these item evaluation and rating summaries form the bulk of Annex 1. The 1-4 rating scale has the following interpretation: 1= clearly inadequate/ not at all helpful 2= barely adequate/ not very helpful 3= adequate/ fairly helpful 4= more than adequate / very helpful. Since the questions in the template are more applicable to research and other analytical work, rather than projects, three of the questions were considered NA (non-applicable) and were not included in the rating of projects. A more appropriate rating criterion for projects is how well, if at all, they have captured experience or incorporated the knowledge found in the analytical work and ESW. Hence, beyond the rating scale, we provide a qualitative evaluation in addition, when reviewing individual projects, on how well they have captured the current state of the art in knowledge on education. Since the "analytical" attribute is the main criterion on which a particular work was rated, it is worth spelling out how it was used in this report. This might be best illustrated by means of two examples at the extreme ends of the scale. · A work was rated non-analytical if it was descriptive in nature, with hardly any tables or statistical analysis generating new knowledge, or if it did not attempt to incorporate cutting edge existing knowledge, or if it contained only cliché generic sentences that sound nice but could be applied to any country in the world, regardless of the specific country's conditions. · A work was rated analytical if it contained information specific to the subject country, e.g. data from a household survey, and/or presented the results of statistical analysis relevant to the issue. Table 1 provides a summary of the results of the evaluation in terms of the overall mean of the scale. (Mean score 2.5). 3 Table 1. Mean Rating Score Item Item score Group score Major research and analytical work 3.6 Diversified Secondary Education 3.4 Educational Performance of the Poor 3.9 Women's Education 3.4 Quasi policy papers 2.8 Education Sector Strategy 3.3 Poverty Reduction Strategy Sourcebook 4.0 Poland Secondary Education and Training 1.1 Major events 3.3 Education for All (EFA) Dakar 2000 2.6 EFA Development Committee 2001 3.6 EFA Development Committee 2002 3.6 ESW 2.3 Brazil Education 2.4 China 21st Century Education 1.7 Pakistan Education Review 2.4 Tanzania Technical and Higher Education 2.6 Projects 2.2 Bangladesh Female Secondary Education 3.5 Brazil Basic Education Quality 4.0 China Basic Education 1.5 Malawi Secondary Education 1.5 Pakistan Northern Education 1.3 Tanzania Primary Education 1.3 Electronic sites 2.9 Education for All 2.1 Economics of Education 3.2 Education Statistics 3.3 Source: Annex 1. 4 III. Major research or analytical work1 There have been several major knowledge milestones at the World Bank that changed the nature of lending for primary and secondary education. Although the first falls outside the 1996 cut-off date of the OED review, it has been included because it had a major influence on the nature of Bank lending between the early 1980s and today. - From a vocational to a general secondary education curriculum. The key analytical work in this respect is a 1985 World Bank publication "Diversified Secondary Education and Development: The Case of Colombia and Tanzania" (Johns Hopkins University Press). Applying an investment in human capital methodology, and using cost-benefit analysis, it was shown, counter intuitively, that a general secondary education curriculum usually offers higher economic returns than a vocational curriculum, as the former exhibits lower costs and greater flexibility. This knowledge was originally generated by the Bank by means of a large research project (known as DiSCuS ­ the Diversified Secondary Curriculum Study) based on field work in Colombia and Tanzania. - From bricks and mortar to software inputs. The key analytical work underpinning this change appeared in a 1992 World Bank publication "Educational Performance in a Poor Country (Oxford University Press). This work contained original knowledge produced by the Bank, with field work in Northeast Brazil. It was shown that the so- called "software inputs", such as reading and writing materials, are more cost-effective in raising school achievement than school buildings. Such findings had a profound influence on the nature of subsequent Bank lending, which increasingly involved projects in which the software components exceeded buildings or other hardware investments. - Women's education. This is another major commitment of the Bank. The major analytical work was "Women's Education in Developing Countries", published in 1993 by Johns Hopkins University Press. The book applied the human capital framework to women's education, and revealed the magnitude of the gender gap in education in several regions, and provided the necessary support for a gender bias in several Bank projects thereafter. 1In addition to the three pieces of research reviewed below, and in Annex 1, another book was identified that has significantly changed the focus of Bank operations in education from post secondary education to primary education. The key influence for this shift was a 1985 World Bank publication "Education for Development: An Analysis of Investment Choices" by George Psacharopoulos and Maureen Woodhall (Oxford University Press). Until then, education projects were treated by the Bank as primarily construction of buildings. This publication, summarizing in essence the state of the art of the economics of education to that date, as it might be applied to developing countries, treated education as a form of investment in human capital that generates an economic rate of return similar to that of investment in physical capital. Applying this human capital framework, it was shown that investment in primary education exhibited (and still exhibits) a higher rate of return relative to other levels of education, and hence should be given priority in World Bank lending. However, since the consultant of this OED review was a co-author of this analytical work, it was not included in the rating. 5 All the above pieces were rated very highly on the analytical content and knowledge they have provided. (Group average = 3.6). IV. Quasi Policy Papers The last full-blown policy paper the Bank had on education was produced in 1981. For reasons having to do with the difficulty of reaching consensus in a large organization, no education policy paper for the sector as a whole has seen the light in more than 20 years. However, several other documents have appeared that, for all practical purposes, could be considered and have acted as policy paper substitutes. - The 1999 "Education Sector Strategy Paper" provides a good stock taking of World Bank thinking in the area, incorporating the knowledge that has been outlined above. - The education chapter of the 2002 "Poverty Reduction Strategy Sourcebook" was selected, as it was designed to incorporate analytical work on the way education projects can contribute to poverty alleviation ­ another major commitment of the Bank. In the opinion of this reviewer, this piece was very successful in fulfilling its purpose, and was the only one that got a rating of 4. - Since Poland was a selected country, but had no primary or secondary education projects, a publication relating to secondary education was identified for review: "Poland, Secondary Education and Training", 2001. This piece was entirely descriptive in nature, and disappointing in terms of its analytical content. On the basis of the criteria used for evaluating items in this review it therefore had a rating of 1. By virtue of the inclusion of Poland, the mean of this group is only 2.8, although the mean score of the other two items is 3.6. V. Major events addressed to the topic - In 1990, the Bank co-sponsored (with Unesco and UNICEF) a major event known as the Jomtien Education for All (EFA) conference, aiming at providing universal primary education by 2000 in all countries that lagged behind in primary school enrolment. Since implementation of such an ambitious target fell behind expectations, the pledge was redefined and renewed in the 2000 Dakar EFA conference, pushing the target date for universal primary education to 2015. - In 2001 and 2002, the Development Committee made a pledge relating to the G8 countries, to contribute $2.5b. for basic education. 6 The theme these events have addressed is at the core of this review. The activities aim at redressing the worst deficiencies in human capital formation in the poorest countries in the world. As such, they deserve a "high" qualitative rating. Moreover, the events were successful in highlighting the importance of education for all and in reaching and influencing donors. This has been demonstrated in the subsequent aid allocations of many donors. However, on the analytical attribute, the initial education-for-all activities, especially the Jomtien conference, missed or grossly underestimated the cost implications of reaching universal primary education coverage. This was somewhat redressed in the 2001 Development Committee report, but there was still too little emphasis on the issue of comparing the cost-effectiveness of alternative strategies for achieving EFA. The group average for these events is 3.3. VI. Economic and Sector Work (ESW) The selection of items to be reviewed was driven by the prior choice of countries by OED. Pakistan and Malawi were added, because of the particular condition of primary education in those countries: - (1998 Bangladesh Post Primary Education) - 1997 Brazil Education - 2000 China 21st Century Education - (1999 Malawi Education Sector Work) - 1997 Pakistan Education Sector - 1996 Tanzania Post Primary Education Out of these, using the Bank's list of ESW, no examples of ESW were found for Bangladesh and Malawi. In both cases, ESW titles are listed in the Bank's data base, but no actual document could be traced by OED. Thus it appears that non-lending support was provided by the Bank, but was not documented in a final report. On its own, this is a "finding" worth OED's attention. Also, the ESW identified for Tanzania (see Annex 2) was published in 1996 instead of 1997, and with a different title. As a whole, the four pieces of ESW that were eventually reviewed, were rated 2.3, i.e., below the scale's average of 2.5. The reason is that they did not contain as much analytical work as they could have done, given the data availability in each country. Actually some of the projects (e.g., Brazil basic education), contain more detailed analytical work than the ESW for that country. Especially disappointing was the ESW for China, which was mainly descriptive. VII. Key projects or reform programs in the area The selection of items to be reviewed was again driven by the prior choice of countries by OED: - 2002 Bangladesh Female Secondary Education Investment Loan 7 - 2001 Brazil Basic Education Program Loan - 1997 China Basic Education Investment Loan - 1998 Malawi Secondary Education Project - 1997 Pakistan Northern Education Project - 2002 Tanzania Primary Education Adjustment Loan Out of the six projects, only two (those for Bangladesh and Brazil) got high marks on the scale, as they contained considerable analytical work. None of the remaining four reached 2 on the scale, in spite of considerable data availability in those countries. The low rating of some of the projects reflects the fact that, to this reviewer, the PADs did not incorporate analysis of data or evidence specific to the country, or even show that the Bank is aware of, or capitalizing on, analytical work previously carried out by Bank staff or others in a particular country. The mean score for this group is 2.2. VIII. Electronic sites In addition to the written material described above, this review evaluated the analytical work that can increasingly be found or be supported by electronic sites. Actually, much of the published material is available on line, including the books. However, the download size of these documents make it impractical for someone with a 56K modem and a Pentium II computer in a developing country to access them easily. Although good for archival purposes, e-versions cannot match the hard copy document when one wants to review it in the manner of this report, and the same is likely to be true for practitioners wishing to access this knowledge for policy making in developing countries. What was missing in particular, is an easily accessible, low MBs synopsis of the mega documents. The review therefore concentrates on the following sites that can be accessed by anyone outside the Bank. They relate to the Education-for-All initiative and contain state of the art information ranging from statistics to issues in the economics of education pertaining to developing countries: - http://www.unesco.org/education/efa/index.shtml - http://www.unesco.org/education/efa/archives.shtml - http://www.worldbank.org/education/economicsed/ Also, an internal site has been reviewed that contains statistics relating to the conditions of education on a global scale: - http://sima/edstats Of these, the economics of education site stands out from the rest. It contains an extremely rich array of analytical material, from inside and outside the Bank, and it is very user- friendly. The Education-for-All sites, though containing useful material, convey the impression of advocacy rather than being based on analysis. 8 The education statistics site goes beyond what one may find in the Bank's "World Development Indicators" CD ROM. However, because of the richness of material, it is sometimes hard to identify the latest series of specific variables. In addition, there are missing data for variables that are crucial to this review, such as net enrolment ratios. The group average for the e-sites is 2.9. IX. Analytical work/ESW vs. project content A major question is the extent of correspondence ­ or lack of it ­ between (a) the findings of research, analytical work and ESW on the one hand, and (b) project content on the other. As a background, Table 2 gives a few basic key statistics on the condition of education in the countries under review. Table 2. Basic Country Statistics Country Per capita Adult Youth Primary Gross Net female incomes female female education primary primary (US$) illiteracy illiteracy pupils enrolment enrolment rate (%) rate (%) female (%) ratio (%) ratio (%) Year 1999 1999 1999 1997 1997 1997 Bangladesh 1484 70.7 60.6 .. .. .. Brazil 7037 15.1 5.9 .. 124.6 .. China 3618 24.5 4.0 47.5 122.7 .. Malawi 586 54.7 40.1 .. .. .. Pakistan 1834 70.0 51.6 39.3 .. .. Poland 8450 0.3 0.2 48.4 .. .. Tanzania 501 34.3 12.2 49.6 66.5 49.1 Source: World Bank Development Indicators 2001, CD ROM. Note: .. missing One should first reconfirm the remark made above regarding education statistics. On the most crucial Bank priorities, i.e. basic education and females, the data are either missing or refer to 1997. Second, given the condition of primary education and illiteracy in Bangladesh and Malawi, as shown by these statistics, it is difficult to justify why the projects have addressed secondary education. It is true that Bangladesh and Malawi need secondary education, as well as university and technical education, just as they also need clean water, sanitation and the like. But given the fact that Education-for-All is a Bank priority, as is girls' education, the 40 to 60 percent illiteracy rate 9 among female youth in Bangladesh and Malawi suggest that another primary education project should have received priority over secondary education. It is true that other donors might be filling the gap in terms of priorities identified by the World Bank, such as "Education for All", or girls' access to primary education, so that it could be justifiable for the Bank to invest in a secondary education project. However, this does not seem to be the case. In Bangladesh, the following agencies are involved: The Asian Development Bank is financing "secondary education reforms", the Norwegian Agency for Development Cooperation is providing "stipends for girls", and the European Union has an operation relating to secondary school teachers. In Malawi, the USAID is concentrating on "primary education and scholarships for girls at the secondary level", and the African Development Bank is involved in access to primary and secondary education , including construction of three secondary schools. In other words, other agencies are mainly involved in secondary education in two countries where access to primary education, let alone its quality, is in an appalling state. Table 3 rates the match between analytical work and project content. For the reasons given above, Malawi and Bangladesh were given a rating of 1 and 2, whereas the other countries got a 4. In particular, the Brazil education project was in full tune with the analytical work reported earlier, i.e. improving primary education quality in the Northeast of Brazil was identified as a priority, and the project aimed at filling this gap. Table 3. The Match between Analytical Work/ESW and Project Content Country ESW-Project Match Score Bangladesh 2 Brazil 4 China 4 Malawi 1 Pakistan 4 Tanzania 4 Overall score 3.2 Score legend: 1 = No match, 2 = Some match, 3 = Adequate match, 4 = Close match 10 X. Conclusion Overall, the Bank is certainly making progress towards becoming a global knowledge institution in the area of education. One can affirm with certainty that the Bank is well ahead in this respect relative to other international organizations that also operate in the area of education. On the positive side, the Bank's research work stands out clearly among the products reviewed. The Bank has sponsored original data collection that relate to education and conducted cutting edge research that has guided field operations. It has also assimilated knowledge produced by universities and other researchers, and put it in a form that can be easily shared with clients and other development partners. However, there is still a divide between the Bank's knowledge base and field operations. Could it be that sometimes projects are driven by "this is what the country wants", without paying attention to existing analytical work? Or that, under operational pressure, staff do not have the luxury to fully absorb and explore with the borrower alternative paths to achieving, for example, the goals of education-for-all? An important implication for OED is the need for closer linkages between Bank sponsored research and analytical work and evaluations of Bank projects. One of the items ranked most highly in this evaluation (item # 3 in Annex 2) was based on research designed to measure the cost effectiveness of alternative policies to improve school quality, using data collected during a Bank project in North-east Brazil. The highest ranking for a project (item #13 in Annex 2) was for a Basic Education project in Brazil that directly used the results of this research. This demonstrates that the Bank's research directly influenced policy making in Brazil. In other countries there are instances where opportunities to link analytical work with Bank projects may have been overlooked. An example is Malawi, where significant changes took place in the 1980s and 1990s in policies on primary and secondary school fees. Some of these changes were related to Bank projects, while others were dictated by political pressures. Research and analytical work on the effects of these changes on demand for schooling and on resource allocation in primary and secondary education might have directly influenced subsequent education policies and projects in Malawi, but there is no evidence that such research took place within the Bank, and in fact no examples of ESW were identified for Malawi. Another implication is that the Bank does not yet pay sufficient attention to dissemination of knowledge to clients and to staff. It is not enough to ensure that up-to-date knowledge is available, in printed format or on e-sites. The commitment to becoming a global knowledge institution and to sharing such knowledge with clients and other development partners requires greater attention to how that knowledge can be most usefully presented to busy politicians, policy makers, Bank and other agency staff. Increasing political and operational pressures on clients and on staff of the Bank and other agencies mean that policy makers and practitioners may not have time to access and absorb relevant analytical work. Training and refresher courses for clients and Bank and agency staff, for example those provided by the World Bank Institute and other organizations, go some way to address this issue, but OED should consider whether even stronger mechanisms are needed to disseminate lessons from World Bank research, analytical work and experience. 11 The above conclusion on dissemination is based mainly on a review of the electronic sites. Although reviewing the Bank's printed work is outside the scope of this report, we believe that the Bank is doing a better job in disseminating its own work by means of its hard copy publications. One should also take into account that the knowledge base commitment is still young (six years old). In future one may see a closer match between what the analytical work suggests, and what the Bank is doing in the field. 12 ANNEX 1. Item Rating ResearchandAnalyticalPapers Item#2inoriginallistinAnnex2 Reference "Diversified Secondary Education and Development: Evidence from Colombia and Tanzania." By George Psacharopoulos and William Loxley. Johns Hopkins for the World Bank, 1985. 337 pages. Access In print. Available at the World Bank Bookstore and can be ordered through any bookseller worldwide. Issue What kind of curriculum should be offered in secondary schools -- vocational or academic? Client / reader Researchers and designers of school curricula. Education policy makers. Document style Mainly academic paper, but also readable by policy makers. Question Comment Response score Was the analytical and advisory work in the document Client clearly identified as Bank 4 relevant to the client's issue? borrowers and Bank staff designing secondary education projects. Education policy makers in general. Was the analytical and advisory work in the document Yes, extremely comprehensive. 3 appropriately comprehensive? Did the document provide the client with the best and Original research in Colombia and 4 most up-to-date knowledge on the issue (including from Tanzania in 1970s and 1980s, building sources outside the Bank)? on past diversified secondary education projects in those countries. Were the ideas and recommendations in the document Ideas stated clearly, although statistical 3 stated clearly? exposition might be beyond the reach of some policy makers. Did the document present information in an objective Document is written from a highly 4 manner (indicating, where pertinent, significant comprehensive perspective, differences that exist in ideas and approaches)? methodologically and regionally. Did the document provide practical advice to decision- The practical advice was not to diversify 3 makers? the secondary school curriculum, a policy that was later adopted by the Bank. Overall appreciation of document Academic, with practical advice. 3 Mean score 3.4 13 Research and Analytical Papers Item # 3 in original list in Annex 2 Reference "Educational Performance of the Poor: Lessons from Rural Brazil." By Ralph Harbison and Eric Hanuskek. Oxford University Press for the World Bank, 1992. 362 pages. Access In print. Available at the World Bank Bookstore and can be ordered through any bookseller worldwide. Issue How to measure school quality and identify the factors that contribute to it in the most cost-effective manner. Client / reader Researchers of school quality. Education policy makers. Document style Mainly academic paper, but also readable by policy makers. Question Comment Response score Was the analytical and advisory work in the Client not identified but can be 4 document relevant to the client's issue? assumed to be Brazil in the first place and other Bank borrowers. Those involved in the design or reform of education policies affecting school quality. Was the analytical and advisory work in the Yes, extremely comprehensive, 4 document appropriately comprehensive? analytically and in terms of advice. Did the document provide the client with the best Ingenious combination of Bank 4 and most up-to-date knowledge on the issue projects and research. (including from sources outside the Bank)? Were the ideas and recommendations in the Ideas stated very clearly on what a 4 document stated clearly? country should do to improve school quality at the basic level. Did the document present information in an Document is written from a highly 4 objective manner (indicating, where pertinent, comprehensive perspective, in a significant differences that exist in ideas and very objective manner. approaches)? Did the document provide practical advice to Practical advice is offered that is 4 decision-makers? highly relevant to improving the quality of primary education. Overall appreciation of document Academic manual, but with strong 3 focus on practical advice. Mean score 3.9 14 ResearchandAnalyticalPapers Item#25inoriginallistinAnnex2 Reference & pages "Women's Education in Developing Countries: Barriers, Benefits and Policies", edited by E. King and M.A. Hill. Johns Hopkins University Press for the World Bank, 1993. 337 pages. Access In print. Available at the World Bank Bookstore and can be ordered through any bookseller worldwide. Issue Women's education lags behind that of men, thus limiting female labor force participation and income. The book examines how educational decisions are made and discusses strategies to improve girls' education. Client / reader Researchers on gender, education and development. Education policy makers. Document style Mainly academic, but also readable by policy makers. Question Comment Response score Was the analytical and advisory work in the Client not identified but can be 4 document relevant to the client's issue? inferred as those involved in design or reform of education policies affecting females. Was the analytical and advisory work in the Yes, as a general overview 3 document appropriately comprehensive? document. Did the document provide the client with the best and Excellent coverage of the state of 4 most up-to-date knowledge on the issue (including the art in gender aspects of from sources outside the Bank)? education, with examples from all regions. Were the ideas and recommendations in the Ideas and recommendations stated 3 document stated clearly? clearly, aiming at improving female education. Did the document present information in an objective Information was presented 3 manner (indicating, where pertinent, significant objectively. differences that exist in ideas and approaches)? Did the document provide practical advice to Excellent chapter 8 devoted to 4 decision-makers? policy advice. Overall appreciation of document Good combination of academic 3 book with focus on practical advice. Mean score 3.4 15 QuasiPolicyPapers Item #4inoriginallistinAnnex2 Reference "Education Sector Strategy". The World Bank, 1999. 80 pages. Access In print. Available at the World Bank Bookstore and can be ordered through any bookseller worldwide. Issue Defining a strategy for World Bank operations in education. Client / reader Bank staff and Bank borrowers. Education policy makers. Document style Down to earth practicality. Readable by the lay person. Question Comment Response score Was the analytical and advisory work in the Very specifically targeted to Bank 4 document relevant to the client's issue? staff and borrowers. Extremely relevant to those involved in design or reform of education systems. Was the analytical and advisory work in the Yes, as a general overview 3 document appropriately comprehensive? document Did the document provide the client with the best Excellent coverage of the issues, 4 and most up-to-date knowledge on the issue although in a necessarily abridged (including from sources outside the Bank)? way. Were the ideas and recommendations in the Ideas stated clearly with many 3 document stated clearly? examples from all Bank regions. Did the document present information in an Document is written from a highly 4 objective manner (indicating, where pertinent, comprehensive perspective, significant differences that exist in ideas and methodologically and regionally. approaches)? Did the document provide practical advice to The prime objective of the paper 2 decision-makers? was to give practical advice ­ and this was done rather well. Overall appreciation of document Good stock taking on the 3 components of an education strategy, adapted to the Bank regions. Mean score 3.3 16 QuasiPolicy Papers Item#26inoriginallistinAnnex2 Reference "PRSP Sourcebook for Education," The World Bank, Draft, June 2001. 58 pages. Access Available only on line. Issue How to use education policy to assist the poor. Education chapter of the Bank's Poverty Reduction Strategy Paper (PRSP). Client / reader Bank staff and borrowers. Document style Analytical and practical. Question Comment Response score Was the analytical and advisory work in the Extremely relevant to the poorest 4 document relevant to the client's issue? countries. Was the analytical and advisory work in the Extremely comprehensive and up to 4 document appropriately comprehensive? date. Did the document provide the client with the best Excellent coverage of the state of the 4 and most up-to-date knowledge on the issue art on what education policies lead (including from sources outside the Bank)? to poverty alleviation. Were the ideas and recommendations in the Ideas and recommendations stated 4 document stated clearly? very clearly. Did the document present information in an Document is written from a highly 4 objective manner (indicating, where pertinent, comprehensive perspective, significant differences that exist in ideas and methodologically and regionally. approaches)? Did the document provide practical advice to The purpose of the paper is to 4 decision-makers? provide practical advice, and this was done very well. Overall appreciation of document Superb combination of analytical 4 work with practical advice. Mean score 4.0 17 QuasiPolicyPapers Item#24inoriginallistinAnnex2 Reference "Poland ­ Secondary Education and Training,". by David Fretwell and Antony Wheeler. Secondary Education Series, The World Bank, No. 22860, August 2001. 7 pages. Access Available from the Bank's Bookshop and on line. Issue Describe state of secondary education in Poland and identify issues relevant for educational reform. The paper is published in a series designed to "expose the complexity of secondary education and training systems and the correspondingly difficult choices that governments face in reforming them" It is not intended to provide analysis or advice, but to contribute to an understanding of general issues in secondary education as a basis for further analysis or research. Client / reader Not specified, but presumably Polish policy makers, administrators and teachers, secondary education policy makers, practitioners and researchers in other countries, and Bank staff. Document style Descriptive, non-analytical. Question Comment Response score Was the analytical and advisory work in the Document does not contain any 1 document relevant to the client's issue? analytical work. Was the analytical and advisory work in the Document extremely terse, due to 1 document appropriately comprehensive? the format of the series. Did the document provide the client with the best No, it did not. It was intended as a 1 and most up-to-date knowledge on the issue brief descriptive summary of (including from sources outside the Bank)? secondary education in Poland. Were the ideas and recommendations in the Recommendations are unclear. 2 document stated clearly? Did the document present information in an No, it did not; it was intended as a 1 objective manner (indicating, where pertinent, brief descriptive summary. significant differences that exist in ideas and approaches)? Did the document provide practical advice to The document contains no practical 1 decision-makers? advice. Overall appreciation of document Could have included some analytical 1 work, or given an more extended summary of existing analytical work. Mean score 1.1 18 Major Events Item # 5 in original list in Annex 2 Reference "Education for All: From Jomtien to Dakar." World Bank, April 2000. 28 pages. Access In print. Available at the World Bank Bookstore and can be ordered through any bookseller worldwide. Issue Support material for the "Education for all" conference held in Dakar, Senegal, April 26-28, 2000. Client / reader Education policy makers. International donors for education. Document style Position/advocacy paper pushing the "education for all" idea. Question Comment Response score Was the analytical and advisory work in the The paper does not contain 2 document relevant to the client's issue? analytical work, as this was not its purpose. The purpose was largely advocacy ­ addressed to both donors and developing countries. Was the analytical and advisory work in the The paper does not contain 2 document appropriately comprehensive? analytical work, as this was not its purpose. The advisory part is overly comprehensive. Did the document provide the client with the best and Yes, the case has been made very 3 most up-to-date knowledge on the issue (including persuasively to advance education from sources outside the Bank)? for all. Were the ideas and recommendations in the Ideas stated clearly although 3 document stated clearly? extremely tersely. Did the document present information in an objective The information was objective, 3 manner (indicating, where pertinent, significant although short on the costs differences that exist in ideas and approaches)? involved to achieve education for all, and who would finance those costs. Did the document provide practical advice to Yes, in the sense of calling for 3 decision-makers? international partnerships to meet the education for all challenge. Overall appreciation of document Advocacy paper mainly addressed 2 to international donors, although advocacy has been consistent with the results of analytical work. Mean score 2.6 19 MajorEvents Item# 22inoriginallistinAnnex2 Reference "Education for Dynamic Economies: Accelerating Progress towards Education for All (EFA)". Development Committee, World Bank and IMF, September 18, 2001. (DC2001-0025). 15 pages. Access Available only electronically. Issue Review of the status of the education-for-all initiative. Client / reader International donors, especially G8 countries. Document style Practical paper signaling the necessity for large financial allocations from donors to secure education for all, and sound policy commitment from the countries concerned. Question Comment Response score Was the analytical and advisory work in the The immediate client of this 4 document relevant to the client's issue? document is the G8 countries and officials in those countries concerned with policies on education and aid, although the ultimate clients are countries that lag in universal primary school coverage. Was the analytical and advisory work in the Yes ­ in summary form ­ although 3 document appropriately comprehensive? the paper was not intended to contain analytical work. Did the document provide the client with the best Excellent treatment of the state of 4 and most up-to-date knowledge on the issue education- for-all, and the reasons (including from sources outside the Bank)? its targets are failing. Were the ideas and recommendations in the Ideas stated extremely clearly. 4 document stated clearly? Did the document present information in an Document is written from a highly 4 objective manner (indicating, where pertinent, comprehensive perspective to serve significant differences that exist in ideas and its purpose. approaches)? Did the document provide practical advice to Practical advice is given to both G8 3 decision-makers? officials and to policy makers in developing countries. Overall appreciation of document Practical paper with sound advice, 3 although objective unrealistic. Mean score 3.6 20 MajorEvents Item# 23inoriginallistinAnnex 1 Reference "Education for Dynamic Economies: Action Plan to Accelerate progress towards Education for All". Development Committee, World Bank and IMF, April 9, 2002. (DC2001-0005/Rev 1). 25 pages. Access Available only electronically. Issue Update of the status of the education-for-all initiative. Client / reader International donors, and especially G8 countries. Document style Practical paper containing an action plan on what donors and countries could do to accelerate the implementation of education for all. Question Comment Response score Was the analytical and advisory work in the The immediate client of this 4 document relevant to the client's issue? document is officials in the G8 countries, although the ultimate clients are countries that lag in providing education for all. Was the analytical and advisory work in the Yes, although the paper was not 3 document appropriately comprehensive? intended to contain analytical work. Did the document provide the client with the best Excellent treatment of the state of 4 and most up-to-date knowledge on the issue education for all, and the reasons (including from sources outside the Bank)? its targets are failing. Were the ideas and recommendations in the Ideas stated extremely clearly. 4 document stated clearly? Did the document present information in an Document is written from a highly 4 objective manner (indicating, where pertinent, comprehensive perspective to serve significant differences that exist in ideas and its purpose. approaches)? Did the document provide practical advice to Practical advice is given to both G8 3 decision-makers? officials and to policy makers in developing countries. Overall appreciation of document Practical paper with sound advice. 3 Mean score 3.6 21 ESW Item # 6inoriginallistinAnnex2 Reference "Brazil -- A Call to Action: Combating School Failure in the Northeast of Brazil," Report No. 18358-BR, December 19, 1997. 61 pages. Access Available from the Bank's reports unit. Also available on line. Issue Identify constraints to improving student outcomes in the Northeast of Brazil. Client / reader Federal Republic of Brazil. Document style Executive summary style based on thirteen research studies that are cited, but not included, in this report. Question Comment Response score Was the analytical and advisory work in the Document does not contain any 1 document relevant to the client's issue? analytical work, but summarizes 13 research studies. Was the analytical and advisory work in the It appears so, although the analytical 2 document appropriately comprehensive? work is not presented in detail. Did the document provide the client with the best By inference, yes. 3 and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the Recommendations were clearly 4 document stated clearly? stated. Did the document present information in an No different approaches were cited 1 objective manner (indicating, where pertinent, in the document. significant differences that exist in ideas and approaches)? Did the document provide practical advice to Practical advice is stated very 3 decision-makers? clearly. Overall appreciation of document Could have included more analytical 3 work, or given more extended summaries of existing studies. Mean score 2.4 22 ESW Item # 8inoriginallistinAnnex2 Reference "China -- Strategic Goals for Chinese Education in the 21st Century," Report No. 18969-CHA, November 30, 1999. 80 pages. Access Available from the Bank's reports unit. Also available on line. Issue Wide ranging document on all aspects of Chinese education. Client / reader People's Republic of China. Document style Document says it is a "Policy note". Question Comment Response score Was the analytical and advisory work in the Document does not contain any 1 document relevant to the client's issue? analytical work. Was the analytical and advisory work in the Document overly terse. 2 document appropriately comprehensive? Did the document provide the client with the best Yes, good annex on educational 3 and most up-to-date knowledge on the issue reforms in former Soviet Union (including from sources outside the Bank)? countries. Were the ideas and recommendations in the Recommendations were clearly 2 document stated clearly? stated, but sometimes are vague. Did the document present information in an No alternative strategies were 1 objective manner (indicating, where pertinent, identified. significant differences that exist in ideas and approaches)? Did the document provide practical advice to Yes, although in a very general way. 2 decision-makers? Overall appreciation of document Could have included some analytical 1 work, or summaries of existing relevant studies. Mean score 1.7 23 ESW Item # 10inoriginallistinAnnex2 Reference "Pakistan -- Towards a Strategy for Elementary Education (Draft)." Report No. 16670-PAK, June 24, 1997. 52 pages. Access Available from the Bank's reports unit. Also available on line. Issue To assist Government to achieve its target of universal primary education by 2006. Client / reader Islamic Republic of Pakistan. Document style Expanded executive summary style. Question Comment Response score Was the analytical and advisory work in the Document does not contain any 1 document relevant to the client's issue? analytical work. Was the analytical and advisory work in the Document overly terse. 2 document appropriately comprehensive? Did the document provide the client with the best Yes, it did. 3 and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the Recommendations were clearly 3 document stated clearly? stated. Did the document present information in an No alternatives were listed. 1 objective manner (indicating, where pertinent, significant differences that exist in ideas and approaches)? Did the document provide practical advice to It did so rather well. 3 decision-makers? Overall appreciation of document Could have included some analytical 3 work, or summaries of existing relevant work. Mean score 2.4 24 ESW Item# 9inoriginallistinAnnex2 Reference "Tanzania -- Technical and Higher Education and Training in Tanzania: Investments, Returns and Future Opportunities." Report No. 15327-TA, January 10, 1997. 65 pages. Access Available from the Bank's reports unit. Also available on line. Issue To inform policy makers in Tanzania on post-primary education and training. Client / reader United Republic of Tanzania. Document style Expanded executive summary. Question Comment Response score Was the analytical and advisory work in the Yes, very relevant. 3 document relevant to the client's issue? Was the analytical and advisory work in the It appears so, although research 3 document appropriately comprehensive? studies on which is based are not included in the document. Did the document provide the client with the best Yes, it did. 3 and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the Recommendations were clearly 3 document stated clearly? stated. Did the document present information in an No alternative approaches are found 1 objective manner (indicating, where pertinent, in the document. significant differences that exist in ideas and approaches)? Did the document provide practical advice to It did so rather well. 3 decision-makers? Overall appreciation of document Could have included some analytical 2 work, or summaries of existing relevant work. Mean score 2.6 25 Projects Item#12inoriginallistinAnnex2 Reference "Bangladesh ­ Female Secondary School Assistance Project-II". Report No:23594- BD, February 7, 2002. 94 pages. Access Available from the Bank's Reports Unit and on line. Issue US$120.9 Bank loan to improve the quality of, and access to, rural secondary education in Bangladesh. Client / reader People's Republic of Bangladesh. Document style World Bank Staff Appraisal Document (PAD). Question Comment Response score Was the analytical and advisory work in the document Very much so. 4 relevant to the client's issue? Was the analytical and advisory work in the document Yes. Used household survey data to 4 appropriately comprehensive? estimate rates of return. Did the document provide the client with the best and most NA -- up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the document NA. -- stated clearly? Did the document present information in an objective Analysis includes alternative cost 3 manner (indicating, where pertinent, significant differences scenarios and their effect on the rate of that exist in ideas and approaches)? return. Did the document provide practical advice to decision- NA -- makers? Overall appreciation of document Good PAD with analytical work not 3 found normally in other PADs. Mean score 3.5 26 Projects Item #13 inoriginallistinAnnex2 Reference "Brazil ­ Ceara Basic Education Quality Improvement Project,". Project Appraisal Document. Report No. 21428 BR, November 29, 2000. 104 pages. Access Available from the Bank's reports unit Also available on line. Issue US$90m Bank loan to promote greater quality, efficiency and equity in the provision of education services in the State of Ceara. Client / reader State of Ceara, Federal Republic of Brazil. Document style World Bank Project Appraisal Document (PAD). Question Comment Response score Was the analytical and advisory work in the Extremely relevant, as a follow-up 4 document relevant to the client's issue? of Harbison and Hanushek (1992) ­ see above (item # 3). Was the analytical and advisory work in the Yes. PAD contains earnings 4 document appropriately comprehensive? functions and rate of return analysis. Did the document provide the client with the best and NA -- most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the document NA -- stated clearly? Did the document present information in an objective Document contains a sensitivity 4 manner (indicating, where pertinent, significant analysis on the rate of return and differences that exist in ideas and approaches)? fiscal estimates Did the document provide practical advice to NA. -- decision-makers? Overall appreciation of document Excellent PAD 4 Mean score 4 27 Projects Item#14inoriginallistinAnnex2 Reference "China ­ Fourth Basic Education Project". Project Appraisal Document. Report No. P7084-CHA, May 6, 1997. 22 pages. Access Available from the Bank's reports unit. Also available on line. Issue US$85m IDA credit to improve access, equity and quality of basic education in some of China's poorer provinces. Client / reader People's Republic of China. Document style IDA Memorandum and Recommendation. Question Comment Response score Was the analytical and advisory work in the Document does not contain any 2 document relevant to the client's issue? analytical work. Was the analytical and advisory work in the Document overly terse. 2 document appropriately comprehensive? Did the document provide the client with the best NA -- and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the NA. -- document stated clearly? Did the document present information in an No alternatives are found in the 1 objective manner (indicating, where pertinent, document. significant differences that exist in ideas and approaches)? Did the document provide practical advice to NA -- decision-makers? Overall appreciation of document Could have included some analytical 1 work, or summaries of existing relevant studies. Mean score 1.5 28 Projects Item #17inoriginallistinAnnex2 Reference "Malawi ­ Secondary Education Project," Report No.17320 MAI, February 28, 1998. 37 pages. Access Available from the Bank's reports unit. Also available on line. Issue US$48.2m IDA credit to expand access and improve quality of secondary education. Client / reader Republic of Malawi. Document style Project Appraisal Document (PAD). Question Comment Response score Was the analytical and advisory work in the Document contains rudimentary 2 document relevant to the client's issue? economic analysis. Was the analytical and advisory work in the Document overly terse. 2 document appropriately comprehensive? Did the document provide the client with the best NA -- and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the NA -- document stated clearly? Did the document present information in an No, it did not. 1 objective manner (indicating, where pertinent, significant differences that exist in ideas and approaches)? Did the document provide practical advice to NA -- decision-makers? Overall appreciation of document Could have included some analytical 1 work, or summaries of existing relevant studies. Mean score 1.5 29 Projects Item #16inoriginallistinAnnex2 Reference "Pakistan ­ Northern Education Project". Report No. P-6959-PAK, September 30, 1997. 20 pages Access Available from the Bank's reports unit. Also available on line. Issue US$22.8m IDA credit to improve the quality of education , access, equity and retention of students, particularly girls. Client / reader Islamic Republic of Pakistan. Document style IDA Memorandum and Recommendation. Question Comment Response score Was the analytical and advisory work in the Document does not contain any 1 document relevant to the client's issue? analytical work. Was the analytical and advisory work in the Document overly terse. 2 document appropriately comprehensive? Did the document provide the client with the best NA -- and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the NA -- document stated clearly? Did the document present information in an No, it did not. 1 objective manner (indicating, where pertinent, significant differences that exist in ideas and approaches)? Did the document provide practical advice to NA -- decision-makers? Overall appreciation of document Could have included some analytical 1 work, or summaries of existing relevant studies. Mean score 1.3 30 Projects Item# 15inoriginallistinAnnex2 Reference "Tanzania -- Primary Education Development Program". Report No. P7466 TA, August 22, 2001. 44 pages. Access Available from the Bank's reports unit. Also available on line. Issue US$150m IDA credit to improve education quality, expand school access, and increase school retention at the primary level. Client / reader United Republic of Tanzania. Document style IDA Report and Recommendation. Question Comment Response score Was the analytical and advisory work in the Document does not contain any 1 document relevant to the client's issue? analytical work. Was the analytical and advisory work in the Document overly terse. 2 document appropriately comprehensive? Did the document provide the client with the best NA -- and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the NA -- document stated clearly? Did the document present information in an No, it did not. 1 objective manner (indicating, where pertinent, significant differences that exist in ideas and approaches)? Did the document provide practical advice to NA -- decision-makers? Overall appreciation of document Could have included some analytical 1 work or summaries of existing relevant work. Mean score 1.3 31 ElectronicSites Items# 18-20,22,23inoriginallistinAnnex2 Reference "Education for All" Access Available externally at: http://www.unesco.org/education/efa/index.shtml http://www.unesco.org/education/efa/archives.shtml Issue Ensuring universal enrolment of primary school age children Client / reader Countries lagging behind in primary school enrolment. International community of donors for education. Document style Web page with many links to background documents and events associated with the education-for-all initiative. Question Comment Response score Was the analytical and advisory work in the Site does not contain analytical 1 document relevant to the client's issue? material or links to it. Was the analytical and advisory work in the Site extremely comprehensive on 2 document appropriately comprehensive? events, but no analytical work. Did the document provide the client with the best Yes, it did. 3 and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the Recommendations were clearly 3 document stated clearly? stated. Did the document present information in an No, it did not. 2 objective manner (indicating, where pertinent, significant differences that exist in ideas and approaches)? Did the document provide practical advice to It did, although the cost implications 2 decision-makers? were not fully analyzed. Overall appreciation of document Could have been more pragmatic and 2 contain more analytical work, particularly on the implications of alternative strategies. Mean score 2.1 32 ElectronicSites Item# 21inoriginallistinAnnex2 Reference "Economics of Education" site Access Available externally at: http://www.worldbank.org/education/economicsed/ Issue Disseminate knowledge on the economics of education for better policy formulation and project design. Client / reader Country education officials. Bank staff. Researchers in the economics of education. Document style Web page with many links to internal and external documents and related sites. Question Comment Response score Was the analytical and advisory work in the Site contains or gives links to rich 4 document relevant to the client's issue? analytical material. Was the analytical and advisory work in the The material presented is very 3 document appropriately comprehensive? comprehensive. Did the document provide the client with the best Yes. It did so in a superb way. 4 and most up-to-date knowledge on the issue (including from sources outside the Bank)? Were the ideas and recommendations in the Recommendations were clearly 3 document stated clearly? stated. Did the document present information in an The site does not cover well critics 2 objective manner (indicating, where pertinent, of the economics of education, as significant differences that exist in ideas and applied by the Bank. approaches)? Did the document provide practical advice to The site contains good practical 3 decision-makers? advice. Overall appreciation of document Excellent site. 4 Mean score 3.2 33 ElectronicSites Item# 27inoriginallistinAnnex2 Reference "Education Statistics" Access Available only at the Bank's intranet: http://sima/edstats Issue Provide up to date statistics on education as a guide for the preparation of education projects. Client / reader Bank staff. Borrowers for education. Document style Database. Question Comment Response score Was the analytical and advisory work in the NA -- document relevant to the client's issue? Was the analytical and advisory work in the NA. -- document appropriately comprehensive? Did the document provide the client with the best Yes, it provided the latest data 3 and most up-to-date knowledge on the issue available on a wide range of (including from sources outside the Bank)? variables. Were the ideas and recommendations in the NA -- document stated clearly? Did the document present information in an Yes, to the extent the data are valid. 4 objective manner (indicating, where pertinent, significant differences that exist in ideas and approaches)? Did the document provide practical advice to NA -- decision-makers? Overall appreciation of document Good statistical base. 3 Mean score 3.3 34 ANNEX 2. Original list of material to be reviewed No. Title Major research or analytical work 1 "Education for Development: An Analysis of Investment Choices" (not included in rating) 2 "Diversified Secondary Education and Development: The Case of Colombia and Tanzania" 3 "Education Performance in a Poor Country" 4 "Education Sector Strategy Paper" 5 "Education for All". Key ESW specific to the selected borrowers 6 1997 Brazil Education 7 1998 Bangladesh Post Primary Education 8 2000 China 21st Century Education 9 1996 Tanzania Post Primary Education 10 1997 Pakistan Education Sector 11 1999 Malawi Education Sector Work Key projects or reform programs in the area 12 2002 Bangladesh Female Secondary Education Investment Loan 13 2001 Brazil Basic Education Program Loan 14 1997 China Basic Education Investment Loan 15 2002 Tanzania Primary Education Adjustment Loan 16 1997 Pakistan Northern Education Project 17 1998 Malawi Secondary Education project Major events addressed to the topic 18 1990 Jomtien Conference on Education for All 19 2000 Education for All Conference in Senegal 20 2002 Development Committee G8 pledge $2.5b for basic education 21 1997 - to date Economics of Education web site Late Additions 22 Development Committee Paper on EFA October 2001 23 Development Committee Paper on EFA April 2002 24 Poland Secondary Education and Training 25 "Women's Education in Developing Countries" 26 PRSP Sourcebook for Education - Draft June 2001 27 Education Indicators Edstats website http://sima/edstats/ 35 ANNEX 3. Terms of Reference -- Expert Knowledge Review In 1996, the World Bank made a commitment to become a global knowledge bank. The commitment signaled the Bank's intention to develop a world ­class knowledge management system and to improve and expand its knowledge sharing with clients and other development partners as a complement to Bank lending. The commitment is based on the notion that the Bank's unique advantage is its ability to combine expert cross-country knowledge with in-depth understanding of country circumstances. To assist OED in assessing the validity of this underlying notion, the consultant will review a sample of the Bank's analytical and project work in the area of primary and secondary education (with a focus on provision of quality education services) to determine the relevance, quality, and innovativeness of knowledge provided by the Bank to its clients. Questions to be addressed in the consultant's review include: 1. Has the Bank's analytical and advisory work addressed the problem in a relevant and appropriately comprehensive way? 2. Has the Bank provided state-of-the art knowledge validated by experience (including from sources outside the Bank)? 3. Has the knowledge provided been practical from the vantage point of decision-makers, program managers, and other key stakeholders? 4. Has the knowledge been provided in a suitably objective way, indicating, where pertinent, significant differences that exist with regard to ideas and approaches. The assessment will be based on: 1. major research or analytical reports 2. key economic and sector work (ESW) specific to selected borrowers 3. three or so key projects or reform programs that reflect the best of the Bank's operational work in the area 4. supporting materials for major events (conferences/workshops) addressed to the topic and newsletters or other forms of ongoing knowledge sharing efforts. In addition, the consultant will review relevant Bank internet-based knowledge and information.