Research Report Series
To implement a task of a high quality, each individual comes up with a plan in advance to realize the established plan. However, one may have to make changes in the implementation process of the plan. This could occur due to unexpected challenges caused by a number of objective and subjective factors during the implementation of a task. Certainly, there are observations and analyses, carried out for understanding these issues in order to base the decisions on the findings of analysis, which sometimes may change the course of actions. A well-researched issue/question conditions an informed decision making process, which is always the guarantee of successfully implemented plan.
The State Students Admission Commission (SSAC) is a centralized state institution responsible for implementing all levels of university admission examinations since 1992. The SSAC designs, implements, and analyzes the results of bachelor, masters, and medical residency exams on an annual basis. Analysis of the exams provide the most complete educational statistics available in Azerbaijan and are used for improvement of examination technology and test questions and general improvement of the education system. The primary objectives of the Unit of Statistic Research, Scientific Projections and Innovations (USRSPI) are in the following directions:
Analysis of data to produce projections prior conducting examinations
Statistical analysis of examination results and test items
Researching innovations in test design
Preparation of applied software required at work.
You may read the research report for the academic year 2014/2015 by downloading the file from the following link: http://tqdk.gov.az/activities/statistics/?EID=1527
USRSPI plays a key role in the overall mission of the SSAC. Before conducting any examination, the USRSPI prepares a project plan covering all phases of the examination process, including preparing technical tasks on the content of test items, providing suggestions on the procedures and rules for organization of examinations, calculating the results of examinations, running a program that distributes placements to higher and specialized education institutions.
Projections are prepared on the basis of research methods. For instance, the SSAC organizes several trial examinations every year. These examinations not only play a role in practice and self-assessment for applicants, but are also important for research purposes. After every trial examination, the SSAC evaluates the functionality of test items and analyzes new test items for inclusion on future examinations. The Unit then drafts new test proposals on the basis of these analyses and plays a large role in shaping the final exams handed out to students.
In addition to analyzing and preparing specific test items, the Unit is also responsible for a broader analysis of the admission plan, applications and the terms of competition. Again, the Unit plays an important role in deciding procedures such as the organization of the examination, realization of the competition, etc. The pre-examination analyses carried out by the Unit plays a vital role in the successful running of the admissions exams. Analyses carried out on specialty groups, education institutions. provide significant information on the situation of social perceptions in education. As a result, a package of conclusions are prepared and sent to the relevant governing authorities for taking into consideration when preparing the next admission plan.
The unit initiates a number of scientific statistical analyses after the completion of examinations. These analyses can be divided into the following groups: analysis after the examination, analysis of test items, analysis of the examination scores, regional analysis, analysis according to education institutions, and analysis of gender aspects. Analysis of admission results, admission plan, and comparison of the admission to specialties of admission programs relative to minimum, medium and maximum scores, and the ranking of higher education institutions on the basis of the examination results are also an important part of the post-examination analyses.
During the analysis of fulfillment of the admission plan both specialty groups and the degree of fulfillment of the plan according to separate education institutions and its reasons are investigated. On the basis of the results of these analyses, suggestions are put forward to be taken into account for the development of the future admission plan. Comparative analysis of the admission to specialties relative to minimum, medium and maximum scores establishes a general perceptions on specialties (well-ranked specialties), which are chosen by better-prepared applicants. Moreover, with the analysis of preferences of specialties, made by applicants in different education institutions, the ranking of higher education institutions of Azerbaijanis prepared. As a result of these analyses, general recommendations and data-based guidance are presented for the consideration of applicants.
The analysis of test items incorporates the following aspects:
the degree of difficulty of test items;
the degree of deviation of test items;
the analysis of distracters (wrong answers);
the analysis of correlation coefficients;
the reliability of test items;
the distribution of correct answers on subjects;
and the comparison of distribution diagrams of applicants, divided into separate sub-groups (gender, age, etc.).
These analyses are carried out both on every test item individually and on the whole test block (booklet). The analyses provide information on the quality of every test item in separate, overall quality of test items and applicants’ mastery of education program on separate subjects. As a result of quantitative analysis, recommendations on development of test items and explanatory information on the current situation of education for each subject are collected.
The quantitative research of examination scores present the distribution of total scores, measures of central tendency (mean, median), dispersion (standard deviation, variance) and association (correlations). These analyses establish an understanding of applicants’ knowledge and abilities, the level of their readiness to college and enable to project the specifications of test items, which should be required according to the format and purpose of each examination.
The unit implements quantitative research per each city and region of the country. At this point, the analysis of applications and specialty groups, indicators on subjects, and admission results are carried out and the distribution scores of applicants admitted to higher education institutions and the earned points are studied. Such analyses assist in understanding the level of education for each region, the applicants’ attitude toward education, tendency of interests to various specialties. In addition, such analyses of regional performance indicators are very useful in comparison of the examination results of applicants. Measures of relative position (percentile, z-score) per each region allow finding out regions performing better than others per each subject and to have data based suggestions and recommendations to present.
Within the framework of the analysis on education institutions, the Unit analyzes each secondary school in the entire country. Here, the analysis of the dynamics on the basis of examination subjects of each education institution and general performance as well as admission indicators and the comparison of examination results with the certificate grades are carried out. The quantitative analysis of education institutions provide sufficient data on the level of education, and the teaching practices for each subject in every secondary school. As a result of the data, presented, education departments of cities and regions learn the possible causes of problems, use these mini reports to understand the situation and think of feasible solutions. It is also necessary to note that this information is available to the public through the website: www.mekteb.edu.az
Besides the analyses mentioned above, research on gender balance of the admission results of examinations is also being carried out. The distribution of scores of male and female applicants and comparative analysis of indicators on examination subjects and general indicators, as well as applicants’ applications according to regions are carried out.
Since 2007, the unit has been implementing a number of investigations in the sphere of application of IRT. The application of the theory enables to implement the comparative analysis of the levels of skills of applicants taking examination in all specialty groups and students participating in final examinations in native language and math.
Every year, all reports of the research and analyses, being done are shared with the public. With this purpose to enlighten the public, The Commission publishes the following journals and booklets:
The journal (“Abiturient” No.12) on scientific statistical analysis of students’ admission results
Booklets on research and analysis of the results of current year graduates’ per each region and city
Gender balance of admission examination results
Research and analysis of admission examination results per subject
Special statistical publications of the Journal
Statistical publications posted online (www.mekteb.edu.az).
It is necessary to note that the results of quantitative researches carried out annually are the subject of discussions in regional conferences held in each regional branch of the SSAC with the participation of involved educational stakeholders.