ICES Journal of Marine Science: Journal du Conseil Advance Access originally published online on August 8, 2009
ICES Journal of Marine Science: Journal du Conseil 2009 66(10):2278-2283; doi:10.1093/icesjms/fsp206
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The value of information in fisheries management: North Sea herring as an example
1 Fisheries and Environmental Management Group, Department of Biological and Environmental Sciences, University of Helsinki, PO Box 65, FIN-00014 Helsinki, Finland
2 Integrative Ecology Unit, Department of Biological and Environmental Sciences, University of Helsinki, PO Box 65, FIN-00014 Helsinki, Finland
3 formerly Cefas, UK, now ICCAT Secretariat, Corazón de María 8, 28002 Madrid, Spain
Correspondence to S. Mäntyniemi: tel: +358 9 191 58710; fax: +358 9 191 58257; e-mail: samu.mantyniemi{at}helsinki.fi
Mäntyniemi, S., Kuikka, S., Rahikainen, M., Kell, L. T., and Kaitala, V. 2009. The value of information in fisheries management: North Sea herring as an example. – ICES Journal of Marine Science, 66: 2278–2283.We take a decision theoretical approach to fisheries management, using a Bayesian approach to integrate the uncertainty about stock dynamics and current stock status, and express management objectives in the form of a utility function. The value of new information, potentially resulting in new control measures, is high if the information is expected to help in differentiating between the expected consequences of alternative management actions. Conversely, the value of new information is low if there is already great certainty about the state and dynamics of the stock and/or if there is only a small difference between the utility attached to different potential outcomes of the alternative management action. The approach can, therefore, help when deciding on the allocation of resources between obtaining new information and improving management actions. In our example, we evaluate the value of obtaining hypothetically perfect knowledge of the type of stock–recruitment function of the North Sea herring (Clupea harengus) population.
Keywords: Bayesian statistics, bioeconomics, decision analysis, stock–recruitment, uncertainty
Received 6 February 2009; accepted 25 June 2009; advance access publication 8 August 2009.
 |
Introduction |
|---|
 Top Introduction The value of informationDiscussionSupplementary materialReferences |
|---|
Fisheries management falls into the category of decision-making under uncertainty. Inherent in such a task is the problem of investing in new information. How much will the new information improve the performance of decision-making, and what is the maximum price that should be paid for new information? The problem of the value of information (VoI) has been recognized and discussed in basic fisheries stock assessment textbooks (Hilborn and Walters, 1992) and journal papers (e.g. Hansen and Jones, 2008a), but examples where the VoI has been explicitly quantified in a fisheries context are scarce. However, McDonald and Smith (1997) provided a tutorial on the basic concepts, and Kuikka et al. (1999) and Moxnes (2003) applied the approach in the context of cod (Gadus morhua) management. Punt and Smith (1999) also evaluated VoI, but neglected parameter uncertainty and relative credibility of alternative model structures.
Quantifying the VoI is more common in fields of decision-making under uncertainty other than fisheries. The concept of the VoI belongs naturally to the theory of information economics, a branch of microeconomic theory (Quirk, 1976). In practical applications, the VoI has been studied, for example, in the context of medical decision-making (Groot Koerkamp et al., 2008; Singh et al., 2008), environmental health-risk management (Yokota and Thompson, 2004a, b), and measuring the importance of space-derived data for resource management (Macauley, 2006).
Our purpose here is to demonstrate the concept of the VoI in a fishery decision-making context. We first introduce the concept with the help of a theoretical example, then provide a more realistic example of the VoI in knowing the functional form of the stock–recruitment relationship (SRR) for North Sea herring (Clupea harengus).
|
The value of information |
|---|
|
TopIntroductionThe value of informationDiscussionSupplementary materialReferences |
|---|
The VoI assigns a numerical value to obtaining a particular form of new knowledge. Most often, the value is understood as a measure of the economic VoI, but there is no need to be so restrictive; any quantitative measure of utility can be used, such as the number of fish landed or a perception of happiness on a scale of 0–100. The theoretical example below illustrates the concept.
An example: the value of perfect information on the location of a fish population
The following example illustrates the calculation of VoI under the framework of Bayesian decision analysis. For more details about Bayesian decision analysis, see, for example, Raiffa (1968) and McAllister and Pikitch (1997). Consider a situation in which a population of 1000 fish moves between two habitats (e.g. offshore and estuary) in such a manner that 80% of the population is always in one of the two habitats. Further, assume that by placing our fishing gear in the same habitat as the fish, we would be able to catch the entire school present in that habitat. Therefore, we have two alternative decisions to be made: DE, fishing in the estuary, or DO, fishing offshore. Furthermore, there are two possible states of nature: FE, where 80% of the fish are in the estuary, and FO, where 80% of the fish are offshore. These alternatives result in four possible combinations leading to different catches, which specify the utility function: C(FE, DE) = 800; C(FO, DE) = 200; C(FE, DO) = 200; and C(FO, DO) = 800.
Our problem is that we have imperfect information regarding the location of the fish stock at the time we are going to fish. Guided by our earlier experience, our expectations about the location of the fish stock are represented by two probabilities: P(FE) = 0.7 and P(FO) = 0.3. The expected utilities of the alternative decisions DE or DO are given as
|
|
What would be the value of a device that could tell us, without error, whether most of the fish population is located offshore or in the estuary, i.e. what would be the maximum price for such a device? The device can be seen as a source of new information providing messages about the resource. Each alternative message potentially has a different value, which could also be zero if the message does not result in changed behaviour. The overall value of the information source is the expected value of the message that would be received. The first step will be to determine the optimal decision in the absence of new information, which in this case is to choose decision DE. The value of each message can then be calculated. In this case, there are two possible messages to be received.
- mO, the fish are offshore. Because the device is assumed to work without error, this information changes the knowledge of the location of the fish, so that P(FE|mO) = 0, and P(FO|mO) = 1. Consequently, the expected utilities of the decisions will change to E(C|DE, mO) = 200 and E(C|DO, mO) = 800, which means that the optimal decision changes to DO. The value of the message is the difference between the expected utilities of optimal decisions based on new information and current information, respectively, i.e. L(mO) = E(C|DO, mO) – E(C|DE, mO) = 800 – 200 = 600.
- mE, the fish are in the estuary. Given this message, the probabilities for the location of the majority of the fish population change to P(FE|mE) = 1 and P(FO|mE) = 0, and the consequent expected utilities are E(C|DE, mE) = 800 and E(C|DO, mE) = 200. Therefore, the optimal decision does not change, and the value of this message is zero, i.e. L(mE) = E(C|DE, mE) – E(C|DE, mE) = 800 – 800 = 0.
As can be seen, the VoI depends on the current uncertainty. This can be examined further by assessing the VoI as a function of P(FE), the probability assigned to the state of nature in which most of the fish are in the estuary. When P(FE) > 0.5, it is optimal to fish in the estuary; otherwise, the optimal choice would be to fish offshore. In this case, the VoI is highest (300) for a person with most uncertainty, i.e. P(FE) = 0.5, and zero for a person who is already 100% sure of the state of nature (Figure 1). Although any new information will obviously be worthless for a person possessing perfect knowledge, it is not necessarily the case that the VoI would peak at 0.5, because the shape of the utility function, along with the uncertainty, also plays a role in decision-making. For a theoretical example about this, see the example about harvesting decisions in Quirk (1976).
|
A more realistic example: the stock–recruitment dynamics of North Sea herring
The purpose of this example is to demonstrate the process of assessing the VoI, or perfect knowledge, on the stock–recruitment dynamics of a fish population. The example population is North Sea herring, as defined by ICES, and we use the same datasets as the ICES Herring Assessment Working Group South of 62°N (HAWG). In our example, though, the goal of the decision-maker is to maximize the expected profit of the herring fishery over a 20-year period by decreasing or increasing fishing pressure relative to the final year in the assessment time-series. Among other things, uncertainty is thought to exist about the type of SRR, for which two alternatives are considered: compensatory and over-compensatory density-dependence in the survival of spawned eggs (see also Nash et al., 2009). These alternatives are represented with Beverton–Holt and Ricker stock–recruitment models, respectively. The idea is to use existing knowledge and data to derive the probabilities for these two alternative states of nature, then to calculate the VoI for a research programme to remove entirely any uncertainty about the true state of nature. In the spirit of the previous example, the research programme can be seen as a device that can send two messages: mB–H, SRR is Beverton–Holt; and mR, SRR is Ricker. The evaluation of the VoI was implemented in the following steps.
- A Bayesian probability model was constructed to describe the population dynamics of North Sea herring. This model included both stock–recruitment functions, with prior probability 0.5, respectively. The model was conditioned on catch and survey datasets covering the period 1960–2003. As a result, posterior probabilities for both stock–recruitment functions were obtained as P(SRR is Beverton–Holt) = 0.43, and P(SRR is Ricker) = 0.57. The results of this model run represent the current state of knowledge. See the Supplementary material for details of the model.
- The same model was fitted a second time to the same dataset, but this time assuming that the stock–recruitment function was known to be either Ricker or Beverton–Holt. The two model runs represent the two hypothetical cases of having perfect information about the form of the stock–recruitment function.
- The population was projected forward for 20 years under each of three knowledge scenarios: existing knowledge and the two stock–recruitment functions. The joint posterior distribution of the state of the population and population dynamics parameters was used as a starting point for the forward projection. Fishing mortality F was assumed to be changed by a multiplier from the last year (2003) of the time-series, then held constant for 20 years. For each knowledge scenario, F-multipliers X = 0.25, 0.5, ..., 5 were used.
- For each combination of knowledge scenario and X, the expected value of the total profit was calculated by discounting future profits annually by 5%. The fishing costs, the price of herring, and the discount rate were taken from a paper by author MR, currently submitted for consideration. As a result, the expected utility could be presented as a function of change in F for each knowledge scenario, and the optimal change could be determined for each case (Figure 2). Based on current knowledge, the optimal choice is X = 1.5; if Beverton–Holt and Ricker were known to be true, the optimal choice would be X = 1.25 and X = 1.75, respectively. The difference between the optimal choices stems from the different shapes of the two functions, which, in turn, will lead to different population projections.
- The expected gain in knowing that the Beverton–Holt function would be an appropriate description of the system dynamics was calculated using the utility function of the corresponding scenario and assessing the difference between the expected utility of the optimal decision with and without the uncertainty:
The same was done for the case of knowing that the Ricker model would be "true":
(1) 
(2)
- The VoI (in Norwegian currency, million NOK) was calculated by averaging the expected gains of both knowledge scenarios, using the posterior probabilities of them as weights.
(3)
|
The results from Step 4 indicate that, regardless of the form of the stock–recruitment dynamics, it would be optimal to increase fishing pressure from the 2003 level. Further, based on the utility functions estimated in Step 4, it is possible to calculate the price of ignoring the uncertainty about the stock–recruitment dynamics, i.e. to evaluate the expected loss if one assumes that the form of the stock–recruitment dynamics was known, when actually it is not. The price is likely to be different for the adoption of different stock–recruitment functions, so this approach could be used to select which stock–recruitment function to use in management advice if it is technically impossible to take account of the actual uncertainty about the form of the stock–recruitment function. Such a price of "overconfidence" can be calculated as follows for both stock–recruitment functions. If the Beverton–Holt stock–recruitment function is assumed to be "true", the optimal change in F would be 1.25, but when uncertainty is taken into account, the optimal change would be 1.5. The expected loss of assuming the Beverton–Holt stock–recruitment function to be correct can now be evaluated by comparing the utilities of these two actions under actual uncertainty (the red line in Figure 2):
|
| (4) |
|
| (5) |
The expected loss through ignoring the uncertainty and assuming the Beverton–Holt function to be the "true" dynamics is smaller than the loss expected if the Ricker function were adopted, so advice based on the Beverton–Holt stock–recruitment function would be the better choice if uncertainty cannot be taken into account.
To conclude in terms of the VoI analysis in this example, it is possible to state that:
- After quantifying the current uncertainty, it is optimal to behave accordingly.
- If the current uncertainty cannot be taken into account for some reason, the next best option would be to assume that the Beverton–Holt dynamics are correct. One would then expect to lose 159 million NOK compared with the case where uncertainty is taken into account.
- Better knowledge of the form of the stock–recruitment dynamics would lead to changes in optimal behaviour, so the information has a non-zero value. The value of perfect information is 240 million NOK, the maximum price to pay for such information. It is also certain that information leading to better, but imperfect, knowledge is valuable, but the value is less than the value of perfect information.
|
Discussion |
|---|
|
TopIntroductionThe value of informationDiscussionSupplementary materialReferences |
|---|
We have demonstrated that a VoI analysis can be performed in a fishery decision-making context when using a complex population model including structural uncertainty. The VoI analysis can be performed on any uncertain quantity that is included explicitly in a probability model which describes the current knowledge of the current state and dynamics of a fishery system. Therefore, the concept should be useful in decision-making at all levels, from an individual fisher to international communities who make decisions and plan and fund research activities. The VoI analysis provides a clear comparison between the consequences of management actions and decisions to obtain more information, because the VoI is expressed on the same scale as the objectives the manager is trying to achieve. Systematic analysis of VoI would provide a way of finding the most critical uncertainties from a decision-making perspective. As we demonstrated here, the steps needed to obtain the VoI also produce results that can be used to calculate the price of overconfidence, the expected loss of ignoring the uncertainty that admittedly exists. Such an approach might be helpful when trying to simplify a complex model structure so that it can be used in practice. As intentional simplification of a model for a system known to be more complex typically leads to overconfidence about the state of the system and model parameters, the process of simplification could be guided by the goal of trying to minimize the price of overconfidence. The simplified model with artificial certainty would still work similarly to the more realistic model. A similar procedure has also been suggested by Morgan and Henrion (1990), who calculated the expected value of ignoring uncertainty.
Although the concept of VoI is very useful as such, its credible quantification requires much multidisciplinary work. The first requirement is the quantification of knowledge of the dynamics and current state of the fishery. This includes the dynamics of the fish population as well as fleet behaviour. To calculate the VoI, current knowledge needs to be expressed as probability statements, implying that a Bayesian approach to stock assessment (McAllister and Kirkwood, 1998) is crucial. In this paper, we considered the potential reduction in structural uncertainty. This uncertainty was taken into account using Bayesian model averaging (BMA; e.g. Hoeting et al., 1999), a method for Bayesian model selection. BMA aims at calculating the posterior probabilities of different structures given prior knowledge and observed data. All models are used simultaneously using the posterior probabilities as weights. Posterior probabilities are related to the Bayes factor, which states the relative weight of evidence (interpreted from data) in favour of one model over another. If prior probabilities of the two models are equal, the Bayes factor is simply the ratio of the posterior probabilities.
Typically, fishing is an economic activity, which means that the objectives of fishery management include, inter alia, economic goals. A challenge is to formulate economic and other goals in a single mathematical function, which can be used to define optimal decisions and evaluate the VoI. It is not, at least traditionally, a scientist's responsibility to determine the goals of management, but it is a scientific task to elicit the preferences of decision-makers and to translate them into a mathematical representation.
The economic part of a system's dynamics is also likely to include great uncertainty (future fuel price, the price elasticity of fish, etc.), and these uncertainties have to be stated in the form of probability statements to evaluate the VoI. One of the results of the analysis could be that surveys to help predict the price of fish would have greater VoI than surveys to estimate the number of fish present.
In addition to demanding practical implementation, the concept of VoI might be philosophically criticized from several perspectives. For example, in our case study of North Sea herring, we assumed uncertainty about the true form of the stock–recruitment dynamics and provided two simple alternatives. In our example, we hypothesized that we could obtain perfect information about which functional form was correct, but strictly speaking, this was impossible because nature is not a mathematical model. Therefore, our analysis is only valid in a "computer game scenario", in which we cast our understanding about a biological process in a mathematical framework where each biological hypothesis is represented by a single mathematical model and where only one is considered true. Of course, the same criticism applies to any type of assessment, including mathematically modelling natural processes; none of the models can be true outside the computer game scenario. It should be noted that VoI for a data-collection programme could be evaluated without the philosophical problem of perfect information. The potential new survey can be included into the probability model, and the VoI evaluated by integrating the posterior distributions obtained after assuming perfect knowledge of the survey data.
Another problem with the concept of VoI may arise from the fact that the value of improved knowledge can only be evaluated based on hypotheses considered to be feasible and included in the probability model. This means that a VoI for research programmes that may generate new hypotheses about causal relationships cannot be quantified. The VoI can be evaluated only for information sources that help in distinguishing between hypotheses considered possible. Of course, the same shortcoming applies to any modelling task; one cannot model something not considered potentially existing when the model is constructed.
Our analysis concentrated on the hypothetical situation that perfect knowledge of the model structure could be obtained. Analysis of imperfect knowledge would proceed with the same logic, but instead of assuming that new knowledge would update the probability of the correct model equalling 1, the probability would be increased based on the assumed likelihood function obtained from the new research (McDonald and Smith, 1997).
It should be noted that VoI does not have one universally true value to be found by science. Instead, it is, by definition, a subjective quantity that measures the VoI perceived by a particular person or group. This is because VoI depends on three subjective elements: (i) knowledge of the current state and dynamics of the system, (ii) the utility function which states the objectives of the decision making, and (iii) the selection of potential management actions considered. Elements (ii) and (iii) typically depend on political decisions, and Element (i) is the result of scientific reasoning. It can be argued that one objectively true value exists for the state of nature at any given point in time, but it is the uncertainty about that true state which inevitably remains subjective. The Bayesian approach for dealing with this uncertainty is to describe the uncertainty with probability statements, where the probability is understood as a personal degree of belief (Ramsey, 1926; Savage, 1954; de Finetti, 1975). Therefore, there are no universally true values for the Bayesian probabilities (Nau, 2001; O'Hagan et al., 2006). There are three kinds of probability statement necessary within the Bayesian stock assessment: the degree of belief about (i) the model parameters and initial state of the system (the prior), (ii) the state of the system, given the previous state (population model), and (iii) the observable data (observation model), given the state of the system (Buckland et al., 2007). All these marginal and conditional probability statements are derived from current understanding of the system. This also includes the observation model, which defines the likelihood function together with the population dynamics model and, thereby, serves as a predefined, subjective logic for interpreting the observations and updating the prior beliefs.
The Bayesian approach has been criticized because it explicitly includes subjective elements. It is generally thought that the use of prior knowledge of the parameters (and about the initial state) renders the analysis subjective, which has led some authors to suggest that the Bayesian analysis could be made objective through using minimally informative, vague, prior distributions (Munch et al., 2005; Berger, 2006). However, the use of a vague distribution does not change the interpretation of the probability as a degree of belief, it only states that "I do not know anything a priori", which is also a subjective statement. The observation and population models are also subjective choices of the analyst, based on current understanding of the system. Therefore, interpretation of the objective facts (the data) is subjective. Of course, this is not a privileged property of the Bayesian approach; the same argument applies to any modelling that includes a likelihood function or other similar human choices (Savage, 1954; Dennis, 1996).
The common recipe of using vague priors is problematic when analysing VoI. If the population dynamics model is based on biological knowledge of the species, the parameters of the model should have a clear biological interpretation. Before seeing any traditional stock assessment data, a fishery expert needs to be able to identify parameter values more credible than other values. If such existing knowledge is intentionally withheld by the use of vague priors in the analysis of VoI, the value of new information is overestimated because current uncertainty is overestimated. The result of analysis would then show that more funds than necessary should be directed to new research. The opposite would happen if variables known to be uncertain were treated as factual (e.g. natural mortality rate M = 0.2). Therefore, it would be in the best interests of those who would pay for the new research that existing knowledge be taken into account in the analysis of VoI as honestly informative priors. To help the evaluation of the extent of prior information included and ignored, we suggest developing standardized ways of reporting the sources of information used in Bayesian stock assessment (see the Supplementary material for an example).
The concept of VoI is potentially most useful when analysed jointly with the value of control (VoC). Analogous to VoI, VoC is the increase in expected utility that would result from obtaining control of a variable that is currently uncontrollable. Comparison of VoI and VoC can be used to distribute limited resources between management actions and gathering new information. These concepts were utilized by Varis et al. (1990), who demonstrated in a water-quality decision analysis that the most risk-averse strategy was to invest mainly in management actions rather than monitoring, because only management actions improved the state of the system. Similar types of analysis have been done in fisheries science, but without explicit consideration of VoI and VoC (e.g. Hansen and Jones, 2008a, b).
|
Supplementary material |
|---|
|
TopIntroductionThe value of informationDiscussionSupplementary materialReferences |
|---|
Supplementary material is available at ICESJMS online to cover the basic stock assessment model for North Sea herring. Among others, two references are used only in the Supplementary material (Carlin and Chib, 1995; Gilks et al., 1996), but for the sake of completeness of referencing of the whole document here, both are listed below.
|
Acknowledgements |
|---|
This study was carried out with financial support from the Commission of the European Communities, specific RTD programme "Specific Support to Policies", SSP-2005-022589 "Precautionary Risk Methodology in Fisheries". It does not necessarily reflect its views and in no way anticipates the Commission's future policy in the area. We acknowledge the input of the two anonymous referees whose comments improved the quality of the paper. Funding to pay the Open Access publication charges for this article was provided by the Fisheries and Environmental Management Group of University of Helsinki.
|
References |
|---|
|
TopIntroductionThe value of informationDiscussionSupplementary materialReferences |
|---|
-
Berger J. The case for objective Bayesian analysis. Bayesian Analysis (2006) 1:385–402.[CrossRef]
Buckland S. T., Newman K. B., Fernández C., Thomas L., Harwood J. Embedding population dynamics models in inference. Statistical Science (2007) 22:44–58.[CrossRef][Web of Science]
Carlin B. P., Chib S. Bayesian model choice via Markov-chain Monte-Carlo methods. Journal of the Royal Statistical Society (1995) 57:473–484.
de Finetti B. Theory of Probability, 1. (1975) Bristol: John Wiley & Sons.
Dennis B. Discussion: should ecologists become Bayesians? Ecological Applications (1996) 6:1095–1103.[CrossRef][Web of Science]
Gilks W. R., Richardson S., Spiegelhalter D. J. Markov Chain Monte Carlo in Practice. (1996) London: Chapman and Hall. 486.
Groot Koerkamp B., Nikken J. J., Oei E. H., Stijnen T., Ginai A. Z., Hunink M. G. Value of information analysis used to determine the necessity of additional research: MR imaging in acute knee trauma as an example. Radiology (2008) 246:420–425.
Hansen G. J. A., Jones M. L. The value of information in fishery management. Fisheries (2008) a 33:340–348.
Hansen G. J. A., Jones M. L. A rapid assessment approach to prioritizing streams for control of Great Lakes sea lampreys (Petromyzon marinus): a case study in adaptive management. Canadian Journal of Fisheries and Aquatic Sciences (2008) b 65:2471–2484.
Hilborn R., Walters C. Quantitative Fisheries Stock Assessment. Choice, Dynamics and Uncertainty. (1992) London: Chapman and Hall. 570.
Hoeting J., Madigan D., Raftery A., Volinsky C. Bayesian model averaging: a tutorial. Statistical Science (1999) 14:382–417.[CrossRef][Web of Science]
Kuikka S., Hilden M., Gislason H., Hansson S., Sparholt H., Varis O. Modeling environmentally driven uncertainties in Baltic cod (Gadus morhua) management by Bayesian influence diagrams. Canadian Journal of Fisheries and Aquatic Sciences (1999) 56:629–641.
Macauley M. K. The value of information: measuring the contribution of space-derived earth science data to resource management. Space Policy (2006) 22:274–282.[CrossRef][Web of Science]
McAllister M. K., Kirkwood G. P. Bayesian stock assessment: a review and example application using the logistic model. ICES Journal of Marine Science (1998) 55:1031–1060.
McAllister M. K., Pikitch E. K. A Bayesian approach to choosing a design for surveying fisheries resources: application to the eastern Bering Sea trawl survey. Canadian Journal of Fisheries and Aquatic Sciences (1997) 54:301–311.
McDonald A. D., Smith A. D. M. A tutorial on evaluating expected returns from research for fishery management. Natural Resource Modeling (1997) 10:185–216.
Morgan M. G., Henrion M. Uncertainty. A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. (1990) Cambridge: Cambridge University Press. 332.
Moxnes E. Uncertain measurements of renewable resources: approximations, harvesting policies and value of accuracy. Journal of Environmental Economics and Management (2003) 45:85–108.[CrossRef][Web of Science]
Munch S., Kottas A., Mangel M. Bayesian nonparametric analysis of stock–recruitment relationships. Canadian Journal of Fisheries and Aquatic Sciences (2005) 62:1808–1821.
Nash R., Dickey-Collas M., Kell L. T. Stock and recruitment in North Sea herring (Clupea harengus); compensation and depensation in the population dynamics. Fisheries Research (2009) 95:88–97.[CrossRef][Web of Science]
Nau R. F. De Finetti was right: probability does not exist. Theory and Decision (2001) 51:89–124.[CrossRef][Web of Science]
O'Hagan A., Buck C. E., Daneshkhah A., Eiser J. R., Garthwaite P. H., Jenkinson D. J., Oakley J. E., et al. Uncertain Judgements: Eliciting Experts' Probabilities. (2006) West Sussex: John Wiley & Sons. 321.
Punt A. E., Smith A. D. M. Harvest strategy evaluation for the eastern stock of gemfish (Rexea solandri). ICES Journal of Marine Science (1999) 56:860–875.
Quirk J. P. Intermediate Microeconomics. (1976) Chicago: Science Research Associates. 359.
Raiffa H. Decision Analysis. Introductory Lectures on Choices under Uncertainty. (1968) MA: Addison-Wesley, Reading. 309.
Ramsey F. P. Truth and probability. In: The Foundations of Mathematics and Other Logical Essays, by Frank Plumpton Ramsey—Braithwaite R. B., ed. (1926) New York: Harcourt, Brace and Co. 156–198. 292.
Savage L. J. The Foundations of Statistics. (1954) New York: John Wiley & Sons. 294.
Singh S., Nosyk B., Sun H., Christenson J. M., Innes G., Anis A. H. Value of information of a clinical prediction rule: informing the efficient use of healthcare and health research resources. International Journal of Technology Assessment in Health Care (2008) 24:112–119.[Medline]
Varis O., Kettunen J., Sirviö H. Bayesian influence diagram approach to complex environmental management including observational design. Computational Statistics and Data Analysis (1990) 9:77–91.[CrossRef]
Yokota F., Thompson K. M. Value of information analysis in environmental health risk management decisions: past, present, and future. Risk Analysis (2004) a 24:635–650.[CrossRef][Medline]
Yokota F., Thompson K. M. Value of information literature analysis: a review of applications in health risk management. Medical Decision Making (2004) b 24:287–298.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||