Safety Reports
Aviation Safety Data Accessibility Study Index:
Analysis and Interpretation of Safety Data
Recent Research
The overwhelming consensus among researchers is that statistical analyses of such measures as death risk, accident rates, and incident rates can say little about differences among carriers, although there are differences between groups of carriers (e.g., major and national carriers vs. regional carriers) and for the aggregate industry over time. All measures of accident and incident rates have declined markedly over time. (This is so even though the absolute numbers of accidents and incidents per year might be stable or even increasing, because of the overall expansion of the industry.) This basic point might be usefully emphasized, because the average person mainly sees information about the frequency and extent of serious accidents, and rarely sees information about aviation's increasing exposure to risk, due to the rapid growth in the number of commercial aviation flights. When an accident occurs, there is an intense period of media coverage and speculation as to what might have caused it. In such an environment, it may be easy to lose sight of how safe air transport has become.
Many researchers have used normalized accident and incident data to analyze the safety of the U.S. commercial aviation industry, including changes in the level of safety over time. Conclusions common to these studies are that the risk of death or serious injury for air travelers is exceedingly small, that this risk fell dramatically between the 1970s and the 1980s, and has remained at these lower levels since then. For example, a passenger who randomly chose a U.S. domestic jet flight between 1967 and 1976 would have a one in two million chance of dying. This death risk fell to one in seven million in the decades 1977-1986 and 1987-1996. Using data from 1990 to the present, the death risk falls to one in eight million. Stated somewhat differently, if a passenger facing a death risk of one in eight million chose one flight at random each day, she would, on average, go for 21,000 years before perishing in a fatal crash. (Hinson, 1996)
While acknowledging that the overall safety record of U.S. commercial aviation is high and has improved over time, researchers have tried to determine whether measurable differences exist in the safety records of individual carriers. There is no evidence found in the accident and incident safety data for individual carriers that allows distinctions to be made between carriers which belong to homogeneous "peer" groups. (GRA 1988, Barnett-Higgins 1989, Oster, et al., 1992, Barnett 1996) This result is illustrated in Exhibit 1, which shows the possible "safety rankings" of U.S. major passenger carriers for three (overlapping) decades. The rankings are based on the death risk for a person who randomly chose one of the airlines' flights during the decade of interest. In each decade, those airlines that suffered no fatalities are given asterisks; they are ranked by number of flights performed. The point to be made is that the rankings are very unstable--the carrier ranked first was different in all three periods, and the airline that was best in one period always fell to the bottom half in the other two periods. It illustrates that because fatal air accidents are so rare among major (and other) U.S. carriers, even airlines with the same safety record over the long-run can have differing accident records over shorter spans of time. Thus, on the measure perhaps most important to a passenger, there are no consistent or persistent distinctions among the major jet carriers.
Exhibit 1
DEATH RISK RANKING FOR TEN YEAR PERIODS
Airline 1984-1993 1979-1988 1974-1983 Airline A 1* 6 7 Airline B 4 4 5 Airline C 5 5 1* Airline D 7 7 2* Airline E 2* 2* 4* Airline F 3 3 8 Airline G 6 1* 6 Airline H 8 8 3* __________
Source: Data adapted from Barnett, 1994.
Top of Page There is probably a natural human inclination to use available numerical information for ranking purposes, and it is always possible to calculate airline-specific accident rates and use them to order the airlines from highest to lowest. However, it is important that such data be analyzed to determine whether observed accident rates among individual carriers are significantly different. When considering the issue of discernible safety differences between individual carriers, it is useful to recall some basic concepts from statistical analysis. Saying that carrier A is less safe than carrier B is saying that a flight by A is more likely to be involved in an accident or incident than a flight by B. If A and B are equally safe, there is no difference in these likelihoods. Unfortunately, individual carrier safety (or aviation system safety in general) is not directly observable. Therefore, differences in safety must be inferred from the statistical analysis of observable data judged to be relevant for safety concerns. The role of statistical analysis is to determine whether observable evidence, such as the actual accident rates achieved by the carriers, is consistent or inconsistent with the presumption that the two airlines are equally safe.
Consider, for example, a six-sided die. If the die is fair, there is an equal likelihood, or probability, that any of the sides will be turned up when the die is rolled. In most cases, however, it is impossible to observe directly that the die is fair. One way to test a die's fairness is to roll it repeatedly. Suppose we roll the die six times. It is possible that a fair die rolled six times could turn up the same number on more than one of the rolls. In fact, for a fair die, with each of the sides equally likely to turn up, the probability of the same number turning up four or more times is around 10 percent, which is to say that, on average, such a result would happen one time in ten. For the purposes of statistical analysis, seeing a die turn up the same number four or more times in six rolls would not be conclusive evidence that the die was unfair. Stated somewhat differently, seeing the same number come up four or more times in six rolls is not inconsistent with a presumption that the die is fair.
It is useful to compare this example with a hypothetical record for U.S. domestic carriers over a period of time. Suppose that in that time period, there were six major jet crashes, and four of them were suffered by Airline A. The safety of individual U.S. air carriers is not observable, but this accident record, seemingly skewed in Airline A's disfavor, is observable. Just as with the example of the die described above, it may not be possible to conclude, based on the accident record, that Airline A must have been less safe than other U.S. domestic carriers. In other words, the evidence available may not allow a statistically significant distinction to be made between Airline A and other carriers. Past research on aviation safety has been unable to find statistically significant differences among individual carriers (within peer groups) based on their accident records. This is due in part to the extraordinarily small number of accidents that do occur.
An important implication of the research results described above is that there is currently no evidence in accident data that would support the ranking of individual airlines based on their safety records, at least for U.S. domestic carriers. While there may be apparent differences in carrier safety records at any particular time, due largely to the infrequent but catastrophic nature of an air accident, there is no evidence that such distinctions persist nor that they are predictive of future safety performance. Rankings of airlines based on past accident records therefore provide no information to consumers seeking to make safety-enhancing comparisons for current or future travel choices.
Some observers, who acknowledge that there is no evidence that would support the ranking of air carriers based on their safety records, would like to consider a "rating" system for informing the public about differences between carriers in safety performance, safety effort, and perhaps other areas. An example of such a rating system, which includes (to a small degree) safety information about carriers, along with other service attributes, to construct a rating scale for airlines can be found in Bowen and Headley (1996). Such an approach is perhaps best left to organizations and firms in the private sector. The role of government is, arguably, to ensure that all carriers meet and maintain common high standards of safety, and to use its regulatory powers to halt deteriorations in safety that might occur at any carrier. If other organizations perceive there is a market for a broader set of information, they can seek to meet that need.
Researchers have had some success in identifying statistically significant safety differences among different groups of carriers. (Higgins 1987, GAO 1988, GRA 1988, Barnett-Higgins 1989, Oster et al. 1992, Stouffer 1992, FAA 1996c, GAO 1996) These studies have found that carriers based in the U.S. and other developed countries consistently have lower accident rates than carriers based in less developed countries and that major U.S. domestic carriers using jet aircraft have lower accident rates than smaller U.S. regional or commuter carriers. Some have also found that established U.S. domestic carriers have lower accident and incident rates than "new entrant" carriers (FAA 1996c, GAO 1996), although there is not agreement among experts that the carrier groupings in these studies are appropriately designed. (There are some concerns about the selection of carriers in each group as well as in the types of events included in the measure of accidents.) It should be emphasized that these studies also conclude that there are no significant differences in risk to life or limb when looking at individual carriers that belong to a homogeneous group of air carriers.
To date there has been relatively little research into relationships between accidents and less serious safety measures such as incidents or surveillance data. New research in this area could provide important findings about the possibility of predicting future accident rates rather than analyzing actual rates after the fact. GRA (1988) examined relationships between accident, incident, and enforcement action rates among major, national, and regional carriers. No relationship was found between incident rates and accident rates. However, it found that there was a statistically significant relationship between accident rates and enforcement action rates among national and regional carriers (but not for major air carriers). However, it was unclear whether higher accident rates led to higher enforcement action rates, or vice versa. An analysis (FAA, 1990) of the relationship between Near Midair Collision incidents (NMACs) and actual Midair Collision accidents (MACs) found that while NMACs were significantly related to the level of airport activity (and congestion), there was not a statistically significant relationship between a particular type of incident (NMACs) and a related type of accident (MACs).
Professor Arnold Barnett of MIT has examined the relationship between passenger death risk and the occurrence of nonfatal safety events, such as incidents and nonfatal accidents, and found them to be negatively correlated for major U.S. airlines between 1990 and early 1996. The results of this research, shown as Exhibit 2, suggest that if anything, passenger death risk is lower on carriers that experience higher rates of negative nonfatal safety events. In fact, as one tries to refine the safety indicator statistic by removing less serious events from it, it actually worsens as an indicator of passenger death risk. This result should not, of course, be taken literally, but it does illustrate the unclear relationships that exist between the statistic most important to a passenger-the risk of dying-with other statistics deemed relevant for aviation safety concerns.
Exhibit 2
CORRELATION OF NONFATAL EVENT STATISTICS WITH PASSENGER DEATH RISK ON MAJOR U.S. AIRLINES
1/1/90 - 3/31/96
STATISTIC (Per 100,000 Flights) CORRELATION Incidents -0.10 Accidents + Incident -0.21 Accidents Only -0.29 Serious Accidents Only -0.34
__________
Source: Professor Arnold Barnett, Presentation to FAA, December 6, 1996.
Top of Page There has been no publicly available research into relationships that might exist between carrier accident rates and the information that is contained in FAA inspection and surveillance reports. Inspection and surveillance data may not be easily comparable for individual carriers, and future research activities in this area should be encouraged. It will be necessary to identify relationships between inspection and surveillance results and observed safety performance before actually using the data for carrier specific comparisons. Surveillance data may also be useful for identifying threats to aviation safety that are currently poorly understood, and there are several cooperative programs for sharing and analyzing data under development in the aviation community. These programs would involve the combined efforts of manufacturers, carriers, and regulators, both within the U.S. and internationally.
Value of Safety Information to FAA
The information generated by accident and incident records, and through FAA's inspection and surveillance activities is valuable because it improves FAA's ability to allocate its limited resources to best serve the safety needs of the flying public. Because most research indicates that safety distinctions between carriers (within homogeneous groupings) cannot be drawn using accident and incident data, inspection and surveillance information is of special importance to FAA. The vast amounts of information contained in inspection and surveillance reports require extensive analysis, and the creation of systems for conducting and disseminating this analysis within FAA is a current and ongoing effort. (FAA 1996a) An example of the use of both accident/incident data and surveillance data can be found in the Department of Defense Air Carrier Analysis System. The Department of Defense, which purchases large numbers of contract flights from civil carriers, uses this system to "score" carriers based on evaluations of their performance in five broad measures of carrier operations-safety, operations, maintenance, financial condition, and service quality. (Ott 1988) However, DOD is a purchaser of contract airlift services and is neither a safety regulatory agency nor a provider of safety information to the public. Some researchers regard the DOD system as overly dependent on heuristic analysis, with insufficient attention given to statistical analysis.
There have been rapid and extensive changes in the U.S. commercial aviation industry since its deregulation in 1978. What was once a highly regulated industry of relatively few stable firms is now a dynamic and complex industry with rapidly changing participants. Many of the changes that have occurred-the development of hub and spoke systems, the increasing sophistication of airline pricing strategies, the rapid entrance (and exit) of new, low-cost carriers-caught many industry experts by surprise.
The primary role for FAA in this changing industry is to ensure that the safety of commercial aviation remains uncompromised amid the turmoil. The collection and analysis of non-accident safety data--incidents and surveillance results-is an essential tool for FAA in this environment. It becomes especially important as FAA's objectives shift from a reactive, learning approach, based on the analysis of past accidents, to a proactive, preventative approach devoted to identifying and remedying potential causes of future accidents.
An important development in FAA's management of aviation safety efforts is the safety partnership program, which will allow FAA to leverage its finite inspector resources. This program will depend on a high degree of cooperation between FAA and established air carriers, and will increase the proportion of safety relevant data that is reported to FAA by established carriers themselves. The implementation of such partnerships will allow FAA to focus its direct oversight efforts on sectors of commercial aviation that could most benefit from more systematic inspection. GAO (1996) identifies new carriers as one class of carriers that should receive more intense surveillance. A recent FAA study (FAA 1996a) also identified the need for more intense surveillance of new carriers and of low cost carriers in general. Other research (FAA 1996c) indicates that low cost carriers might also receive higher levels of surveillance.
It should be noted that more intense levels of surveillance for new and low cost carriers does not necessarily imply that such carriers operate with safety standards that are less stringent than established carriers. It could equally indicate that new carriers, most of which compete strongly on price, are relatively unknown quantities for FAA. Even though a new carrier may have met all certification requirements, FAA will still know less about the operations and management of a new carrier than it does about the activities of a more established carrier. Especially in an era of constrained federal budgets, it may be most efficient, from a safety standpoint, to concentrate FAA's direct surveillance resources on newer carriers, which face greater financial and managerial uncertainties than an established carrier.
Top of Page Value of Safety Information to Consumers
Consumer demand for safety information is a demand for information about the integrity of the system that provides air safety, in whole and in its constituent parts. There is a high degree of consumer interest in this topic which may be at the heart of increased calls for FAA to disclose safety information. Public concerns about aviation safety, as expressed in recent polling results, are presented in Attachment A below.
Just as FAA must adapt itself to a rapidly changing industry, consumers are faced with an increasingly confusing and unfamiliar commercial aviation market. While it may be relatively easy for consumers to make decisions about price and schedule choice, it is more difficult for them to evaluate safety in a changing industry. As increasing numbers of consumers fly, there will likely be increased demand for information about the workings of the aviation safety system and about the status of individual participants in that system.
In these circumstances, the proper role of government and FAA may be to take the lead in providing consumers with information about the high level of safety in aviation, both in aggregate and with respect to individual carriers. Consumers are often trying to inform themselves about new carriers offering services in specific markets, and one role for FAA may be simply to provide information on individual carriers to interested consumers. FAA currently fields many telephone inquiries about specific carriers, an indication that there may be a need for a systematic way of providing such information.
Many consumers are likely to get access to any new information released by FAA through intermediaries such as the media or consumer groups who may put their own interpretation on the information. While there is nothing that FAA can (or should) do about this, it is something to keep in mind. The more user friendly and transparent FAA's presentation of safety information is, the more likely it will be that consumers use the information directly, rather than rely exclusively on intermediaries. Clearly presented information will also make it easier for intermediaries to use safety information responsibly.
A significant component of public concern about flying may stem from the near complete loss of control that a passenger experiences; once a flight commences, there is little a passenger can do to affect the risk of an accident. Risk management and risk communication research has identified such loss of control as an important determinant of individual attitudes toward particular types of risk. This factor is compounded by the catastrophic nature of an air accident, however unlikely, including both the high likelihood of death if an accident occurs and the nature of that death. In addition, commercial air accidents can have hundreds of victims, which adds to the notoriety of an air disaster. The salience of these factors in the psyche of a potential passenger means that an air accident is perceived not as a mechanical or human failure resulting in loss of life, but as a disaster that must have had some identifiable cause and that could have been prevented had appropriate actions or precautions been taken. As flying becomes more common in everyday life, a larger part of the population may hold these concerns.
Because a large component of the public perception of aviation risk may not be easily assuaged by quantifiable risk ratios and accident rates, any communication system intended to inform and reassure the public about safety probably has to address more than the likelihoods of various outcomes and events. Education about air safety and about the structure and reliability of the systems in place to ensure it has to become an ongoing effort. It is also the case that the media will always provide substantial amounts of coverage for particularly catastrophic aviation accidents. The best time to present the public with statistical arguments about aviation safety may not be immediately following an accident. Such arguments are unlikely to be well received at that time, and making them may present FAA more as a promoter of the aviation industry and less as a promoter of aviation safety. It may be that FAA should concentrate its efforts on providing information about aviation safety on a regular and frequent basis. The presentation of specific information and/or reassurance about safety in the aftermath of an accident may be more properly left to the aviation industry.
If people are particularly averse to airliner accidents (as they seem to be) then it is necessary to ask whether the current safety investment and regulatory criteria respond to this level of risk aversion appropriately. Existing criteria for analyzing the benefits and costs of FAA investment and regulatory programs assume expected value decision making. That is, individuals are assumed to be willing to pay up to the expected sum of avoided losses to avoid an air accident. (This means that all other factors equal, society would place the same value on the loss of 100 lives, each lost in one of 100 separate auto accidents, as 100 lives lost in a single airplane accident.) If people are especially risk averse toward aviation accidents with a potential large loss of life in each accident, and no risk premium is placed on airline accidents compared to other accidents, then there may actually be an under-investment in air safety from the perspective of what society really wants. Perhaps in response to the strong public concerns about air safety, the DOT and FAA have established an objective of zero accidents. Since the level of risk in commercial aviation is already so small, it may be informative to communicate the likely expense, in dollars or inconvenience, of further reductions in that risk, since additional funds and energy devoted to aviation safety probably will increase the cost of flying and cause a reduction in funds and energy directed at other social goals. (Keeney, 1995)
Consumers may seek aviation safety information with the hope of making more informed choices among carriers. In this circumstance, safety information is cast in the role of predicting the likelihood of future accidents. If, as most research indicates, a passenger's risk of dying does not significantly differ among the group of carriers from which she must choose, then knowledge about those carrier's accident rates may convey no additional information to the consumer that is relevant for choosing among airlines. Knowing this might in itself be reassuring for consumers. FAA could end up providing information to the public which says that, in most cases, there are no differences in accident rates among available air carriers. Oster et al. (1992) argue that the set of carriers from which a consumer chooses is mainly affected by the kind of market that the consumer is traveling in, and that there are no significant differences in risk among carriers serving a particular market. For example, if all carriers in the New York to Chicago market are major U.S. airlines and these carriers do not have significantly different accident rates, then differences in safety should not affect carrier choice.
Because researchers have been able to find some statistically significant differences in safety between general groupings of carriers, further research into the appropriate design of such comparisons and updating existing results is warranted. However, as Oster et al. (1992) point out, consumers generally do not choose between carriers in different groups. For example, a passenger traveling between two distant cities rarely chooses between a jet carrier and a turboprop carrier, and a passenger traveling between two nearby cities does not always have the option of choosing a jet carrier. Past investigations by the Antitrust Division of the Department of Justice have shown that jet and commuter airlines seldom serve the same city-pair markets, and are not likely to enter many of the same markets. (Bingaman, 1996) For markets served primarily by commuter carriers, therefore, the relevant safety comparison may be not between commuter carriers and major domestic jet carriers, but between commuter carriers and non-aviation transport, such as automobile, bus, or rail.
Top of Page Tensions Between FAA and Consumer Needs for Information
Because some safety data, especially incident data, is self reported by carriers, some thought should be given to how these data are developed and what they represent prior to releasing such data or using it as a basis for comparisons between carriers. A carrier that diligently reports incidents could be made to look relatively prone to these events. Such a carrier may then have an incentive to become less thorough in its reporting practices. This is especially important because some self reported data is relevant for analyzing the safety of other parts of the aviation system. For example, a crowded airport might have a disproportionate number of (carrier reported) near misses, and such data is probably important for analyzing the need for improvements to that airport. A reduction in self reporting by carriers could thus affect not only FAA's evaluation of carrier safety, but also its evaluation of airport safety.
The public availability of carrier reported data has become increasingly important because of FAA's use of safety partnership relationships with established carriers. These partnerships, which place greater self monitoring responsibilities on carriers, necessarily rely on high degrees of trust between FAA and its safety partners. Improper or immature public release of heretofore confidential safety information could have a chilling effect on these relationships. This is an important consideration for FAA because it has found its resources increasingly constrained as the industry becomes more complex, and safety partnerships could be an effective means of leveraging existing inspection resources to allow more direct monitoring of other carriers. Public release of such self reported data may provide consumers with only marginal benefits, at the cost of damaging FAA's ability to fulfill its safety responsibilities.
Surveillance and inspection data has also increased in importance as FAA has moved from a reactive posture, responding to accidents and other safety events, toward a more proactive posture, which seeks to prevent future accidents by identifying their possible causes beforehand. Such data is most effectively gathered and analyzed in an atmosphere of cooperation between FAA and carriers, and several commenters on the initial draft of this report spoke of the chilling effect that public release of surveillance data would have on the level of cooperation between FAA and private participants in the aviation safety system.
The effect of FAA's supervisory efforts and interventions on the level of aviation safety is also an important consideration. It is arguable that the role of FAA is to prevent the development of relationships between "safety indicators" and accident rates. For example, some, but not all research finds a weak correlation between airline profitability and safety. (Rose 1990,1992) Whether or not there is an underlying relationship between financial performance and carrier safety, the effectiveness of increasing surveillance when a carrier is having financial difficulties (or any other operational problems ) may prevent direct observation of any relationship between safety indicators such as profitability and actual accidents. This "intervention" model of regulatory effectiveness might imply that FAA's ability to acquire and act on surveillance and inspection data is essential to its ability to maintain aviation safety. To the extent that public release of surveillance data compromises FAA's abilities to acquire relevant data on carrier operations, releasing that data will be costly and could compromise aviation safety. This raises a subsidiary question of whether the public should be notified of the carriers that are placed under intensified surveillance. If a carrier is placed on the increased surveillance list because of poor financial performance, public knowledge of this may only exacerbate financial distress because passengers may tend to choose other carriers for their air transportation. If it is believed that the heightened surveillance will work to mitigate any negative safety outcome, then the system may be working and disclosure of such information may not be necessary to achieve the desired outcome.
An argument for disclosure of the carriers placed under increased surveillance is that FAA does upon occasion make public information about which carriers are under additional surveillance. (FAA did this after the series of USAir accidents and after the ValuJet accident.) Therefore, if FAA is going to disclose after an accident that the carrier(s) involved had been under heightened surveillance, then perhaps it should routinely report carriers that are under increased surveillance. Otherwise, it may create the impression that such information was being hidden from the public, and came to light only because of the accident.
In the near term, there needs to be a greater understanding among FAA, the airlines, and the public about why FAA may choose to increase the level of surveillance for a particular carrier. Once this is done, FAA should investigate whether the release of such information on a routine basis is warranted and/or whether there is a need for changes in the criteria leading to increased surveillance of an airline (for liability or other reasons). FAA and industry must also consider that making this information available only after an accident or incident can create the impression that the information had been withheld.
A policy of public release of inspection results and surveillance data raises a host of interesting issues. For example, how would it affect the relationship between FAA inspectors and airline personnel? Would it provide positive incentives for carriers to increase compliance with regulations? Does regulatory compliance in and of itself represent an appropriate measure of airline safety? Would the routine release of such information increase or decrease public confidence in the safety of air transportation? Clearly, these issues need to be explored in more detail before reaching a decision about the public release of these data.
Click here to return to Aviation Safety Data Accessibility Study Index