Copyright is owned by the Author of the thesis.  Permission is given for 
a copy to be downloaded by an individual for the purpose of research and 
private study only.  The thesis may not be reproduced elsewhere without 
the permission of the Author. 
 
  
 
 
 
Design and Evaluation of Mass Evacuation Support Systems 
Using Ontologies for improved Situation Awareness 
 
A dissertation presented in partial fulfilment of the requirements for the degree 
of  
 
Doctor of Philosophy   
in  
Information Sciences  
 
at Massey University, Albany,  
New Zealand  
 
 
 
 
Yasir Javed  
 
2012 
  

1 
 
Abstract 
Large-scale emergencies, such as tsunamis, are managed by several teams, e.g. 
emergency managers, military, police, fire services, health care professionals, etc. Close 
co-ordination within and between teams is essential since the failure of a single link can 
risk the whole operation, for example, the mass evacuation of a city or region. Decision-
making in such emergencies is necessarily complex as the situations are dynamic, 
unfolding rapidly, and invariably stressful. Computerised decision support systems can 
facilitate and improve co-ordination and decision making by presenting, structuring, 
processing, and interpreting huge amounts of information in a short span of time.  
However, the power of such systems is enhanced even further if they are designed to 
improve the situation awareness (SA) of individual managers, their shared situation 
awareness (SSA), and team situation awareness (TSA). The goal is to ensure that team 
members have a comprehensive understanding of the situation, not just for their 
individual roles but also for the roles of their colleagues. The aim of the thesis is to 
design a computer based information system to support SA, SSA, and TSA of 
emergency managers for effective decision making and collaborative task performance. 
The thesis describes elicitation of the information requirements for various emergency 
management roles during a mass evacuation using a cognitive task analysis technique. 
Based on the requirements, it explains the design and development of a computer based 
system dubbed Situation Aware Vigilant Emergency Reasoner (SAVER) using 
ontologies for situation assessment and reasoning.  
It is demonstrated that ontologies can be used to classify the SA information since they 
can model the situations in detail and allow the inference on rules and axioms. Ontology 
based reasoning successfully provided the automatic situation assessment according to 
the SA levels. The thesis also details the evaluation of SAVER by measuring SA, SSA 
and TSA of emergency managers using Situation Awareness Global Assessment 
Technique (SAGAT) in simulated mass evacuation scenarios. The evaluation 
demonstrated the superior performance of the computer based system for improving 
SA, SSA and TSA of emergency managers. Moreover, the user interfaces of SAVER 
were also evaluated positive for the human computer interaction (HCI) parameters such 
as usability, ease of use, understandability, learnability, functionality, etc.  
2 
 
Acknowledgements: 
In the name of Allah, the most Gracious and the most Merciful. 
First and foremost, I would like to praise and thank Allah Almighty, Who has granted 
me the grace, courage, and blessing to start such a huge research endeavour and 
complete it.  
I am deeply indebted to my chief supervisor, Professor Tony Norris, whose help, 
creative suggestions and motivation helped me throughout my research project. His 
profound knowledge and experience have been of great value to this project and my 
research training.  I would also like to thank my co-supervisor, Professor. David 
Johnston, for his useful comments and feedback.  
I want to express my gratitude to many people for their contribution, assistance, 
support, and guidance, especially Clive Manley (Auckland Civil Defence Controller and 
Manager of Civil Defence and Emergency Management for Auckland Council), Greg 
Holland, David Neil, Thomas Harre, Jamie Richards from Civil Defence Emergency 
Management (CDEM), who helped me with data collection and understanding the 
emergency management environment of New Zealand. I also want to thank Anuradah 
Mathurani and Ian Bond for advice and useful feedback on my initial research proposal.   
I wish to thank my family, especially my parents and parents-in-law who prayed for my 
success and gave me great strength and confidence by sending me so far from them to 
achieve this milestone. I owe my deepest gratitude, respect, and affection to my loving 
wife and cute daughters, Imaan and Inaya, for their patience and continuous support, 
without which this research was impossible. I am also thankful to all my brothers and 
sister for their prayers and belief in me. I thank all my friends for keeping me energetic 
and lively throughout this research. I would also like to show my gratefulness to Sonya 
Eastmond, Annette Warbrooke, and Freda Mickisch for their continuous administrative 
and moral support.  
 
3 
 
Table of Contents 
Abstract??................................................................................................................................. 1 
Acknowledgements: ..................................................................................................................... 2 
List of Figures ............................................................................................................................. 10 
List of Tables.. ............................................................................................................................ 13 
List of Publications .................................................................................................................... 14 
INTRODUCTION ................................................................................................................... 16 
1.1 Aim of the Research ....................................................................................................... 16 
1.2 Research Contributions .................................................................................................. 17 
1.3 Theoretical Framework .................................................................................................. 18 
1.3.1 Emergency Management overview ....................................................................... 18 
1.3.2 Decision Making in Emergency Management .................................................... 20 
1.3.3 Importance of SA / TSA and SSA in Decision Making .................................... 21 
1.3.4 Ontology based SA .................................................................................................. 22 
1.4 Research Context ............................................................................................................ 24 
1.5 Research Objectives and Questions ............................................................................. 25 
1.6 Research Approach ......................................................................................................... 25 
1.7 Scope of the Research Enquiry ..................................................................................... 26 
1.8 Overview of Thesis ......................................................................................................... 27 
1.8.1 Chapter 2: Theoretical Foundation and Literature Review ............................... 27 
1.8.2 Chapter 3: Research Methodology ........................................................................ 28 
1.8.3 Chapter 4: SAVER: Situation Aware Vigilant Emergency Reasoner ............... 28 
4 
 
1.8.4 Chapter 5: Improving Individual Situation Awareness Using SAVER ........... 29 
1.8.5 Chapter 6: Improving Shared and Team Situation Awareness Using SAVER
 .............................................................................................................................................. 30 
1.8.6 Chapter 7: Discussion ............................................................................................. 30 
1.8.7 Chapter 8: Summary, Conclusions and Future Work ........................................ 30 
THEORETICAL FOUNDATION AND LITERATURE REVIEW ............................ 32 
2.1 Introduction ..................................................................................................................... 32 
2.2 Emergency Management ................................................................................................ 33 
2.2.1 Emergency management in the context of mass evacuation ............................ 34 
2.2.2 Mass evacuation in case of tsunami ...................................................................... 37 
2.2.3 Emergency management in New Zealand ........................................................... 38 
2.3 Emergency Decision Making ......................................................................................... 40 
2.3.1 An overview .............................................................................................................. 40 
2.3.2 Models of emergency decision making ................................................................. 43 
2.3.3 Intuitive or Naturalistic Decision Model.............................................................. 45 
2.3.4 Rational or Analytical Decision Model ................................................................. 46 
2.4 Situation Awareness (SA) ............................................................................................... 48 
2.4.1 SA definition ............................................................................................................. 48 
2.4.2 Endsley?s three level model of SA ......................................................................... 49 
2.5 Situation Awareness in Collaborative Systems ............................................................ 51 
2.5.1 Shared Situation Awareness (SSA) ........................................................................ 51 
2.5.2 Team Situation Awareness (TSA) ......................................................................... 52 
 
5 
 
2.6 Information System Support for Emergency Management ...................................... 55 
2.6.1 Emergency information systems frameworks ..................................................... 56 
2.6.2 Emergency management information systems (EMIS) ..................................... 57 
2.6.3 EMIS support for SA .............................................................................................. 58 
2.7 Ontology ........................................................................................................................... 61 
2.7.1 Definition .................................................................................................................. 61 
2.7.2 Ontology-based situation awareness (SA) ............................................................ 62 
2.7.3 Existing ontology-based systems to support situation awareness (SA) ........... 63 
2.8 Summary and Conclusions ............................................................................................. 64 
RESEARCH METHODOLOGY .......................................................................................... 67 
3.1 Introduction ..................................................................................................................... 67 
3.2 The Philosophical Grounding of Design Science Research ..................................... 67 
3.3 Rationale for Choosing the Design Science Research ............................................... 69 
3.4 Design Science Research Cycles.................................................................................... 70 
3.4.1 The relevance cycle .................................................................................................. 70 
3.4.2 The rigour cycle ....................................................................................................... 71 
3.4.3 The design cycle ....................................................................................................... 73 
3.5 The Design Science Research Methodology (DSRM) ............................................... 74 
3.5.1 Awareness of problem / opportunity ................................................................... 74 
3.5.2 Suggested solution to the problem ........................................................................ 76 
3.5.3 Design and development ........................................................................................ 78 
3.5.4 Evaluation ................................................................................................................. 82 
3.5.5 Conclusion ................................................................................................................ 84 
6 
 
SAVER: SITUATION AWARE VIGILANT EMERGENCY REASONER ............... 86 
4.1 Introduction ..................................................................................................................... 86 
4.2 Problem Identification .................................................................................................... 87 
4.3 Suggested Solution to the Problem ............................................................................... 88 
4.4 Design and Development .............................................................................................. 89 
4.4.1 System requirements elicitation ............................................................................. 89 
4.4.2 The SAVER system .............................................................................................. 105 
4.5 Evaluation ...................................................................................................................... 118 
4.5.1 Scenarios for evaluation ....................................................................................... 118 
4.5.2 Simulated output from SAVER .......................................................................... 119 
4.6 Conclusions ................................................................................................................... 124 
IMPROVING INDIVIDUAL?S SITUATION AWARENESS USING SAVER ...... 126 
5.1 Introduction .................................................................................................................. 126 
5.2 Problem Identification ................................................................................................. 126 
5.3 Suggested Solution to the Problem ............................................................................ 128 
5.4 Design and Development ........................................................................................... 129 
5.4.1 Contextual SA information ................................................................................. 130 
5.4.2 Ontology-based inference for contextual SA information ............................. 133 
5.4.3 Human computer interaction (HCI) design ..................................................... 137 
5.4.4 SAVER features .................................................................................................... 138 
5.5 Evaluation ...................................................................................................................... 141 
5.5.1 Direct measurement of SA using SAGAT ........................................................ 142 
5.5.2 Experiment results ................................................................................................ 148 
7 
 
5.5.3 Human Computer Interface (HCI) evaluation of SAVER.............................. 157 
5.6 Summary and Conclusions ........................................................................................... 163 
IMPROVING SHARED AND TEAM SITUATION AWARENESS USING 
SAVER??.. ........................................................................................................................... 167 
6.1 Introduction ................................................................................................................... 167 
6.2 Problem Identification .................................................................................................. 168 
6.3 Suggested Solution to the Problem ............................................................................ 170 
6.4 Design and Development ............................................................................................ 171 
6.4.1 SSA and TSA requirements elicitation ............................................................... 171 
6.4.2 Ontology based SSA and TSA oriented system design .................................... 178 
6.5 Evaluation ....................................................................................................................... 184 
6.5.1 Experiment design ................................................................................................. 184 
6.5.2 Experiment results ................................................................................................. 188 
6.6 Summary and Conclusions ........................................................................................... 199 
DISCUSSION .......................................................................................................................... 202 
7.1 Introduction ................................................................................................................... 202 
7.2 Formalising SA, SSA and TSA Definitions ............................................................... 202 
7.3 Information Requirements for SAVER Design ....................................................... 207 
7.4 Ontology based SA, SSA and TSA ............................................................................. 208 
7.5 Improving SA using SAVER ....................................................................................... 211 
7.6 Improving SSA using SAVER ..................................................................................... 213 
7.7 Improving TSA using SAVER .................................................................................... 215 
7.8 Relationship Between Individual SA and Various Models of TSA ....................... 216 
8 
 
7.9 Implementation Requirements of SAVER ............................................................... 218 
7.9.1 Organisational requirements ............................................................................... 218 
7.9.2 Architecture requirements ................................................................................... 218 
7.9.3 Graphical user interface (GUI) requirements ................................................... 222 
7.9.4 Testing requirements ............................................................................................ 223 
7.10 Summary ...................................................................................................................... 224 
SUMMARY, CONCLUSIONS ............................................................................................ 228 
AND FUTURE WORK ........................................................................................................ 228 
8.1 Research Questions Answered ................................................................................... 228 
8.1.1 How to design a computer based system for automatic situation assessment 
based on situation awareness (SA) requirements in emergency situations? ........... 228 
8.1.3 How to design a computer based system supporting shared and team SA 
(SSA and TSA) of the emergency managers in mass evacuation operations? ....... 231 
8.2 Future Work .................................................................................................................. 232 
8.2.1 Option Awareness ................................................................................................ 232 
8.2.2 Agents and Trust................................................................................................... 233 
8.2.3 Integration of various disaster management sectors; health, fire, police ...... 234 
8.2.4 Training of emergency managers ....................................................................... 234 
8.2.5 Extension to mobile technologies ...................................................................... 235 
8.2.6 Business decision making .................................................................................... 235 
8.2.7 Military applications ............................................................................................. 235 
References ............................................................................................................................ 237 
APPENDIX A ........................................................................................................................ 267 
Research Information Sheet .............................................................................................. 267 
9 
 
APPENDIX B .......................................................................................................................... 269 
Participation Consent Form ............................................................................................... 269 
APPENDIX C .......................................................................................................................... 270 
Low Risk Research Approvals ........................................................................................... 270 
APPENDIX D ......................................................................................................................... 272 
General Interview questions .............................................................................................. 272 
APPENDIX E ......................................................................................................................... 273 
Templates Used for Information Requirements Gathering .......................................... 273 
APPENDIX F .......................................................................................................................... 278 
Screenshots of the Ontologies Developed ...................................................................... 278 
APPENDIX G ......................................................................................................................... 283 
Simulation Experiment Questions .................................................................................... 283 
Part A: ............................................................................................................................... 283 
EVALUATION .............................................................................................................. 283 
Part B: ............................................................................................................................... 288 
EVALUATION .............................................................................................................. 290 
 
 
 
 
 
10 
 
List of Figures 
Figure 1.1: Emergency management cycle ???????????.?.????19 
Figure 1.2: Flow of thesis chapters ??????????????.?.????31 
Figure 2.1: Phases of evacuation ????????????????.????35 
Figure 2.2: Threat of a catastrophic tsunami on coastal areas ?????.????.37 
Figure 2.3: CIMS Structure ??????????????????.????39 
Figure 2.4: Information sources of SA??????????????.????50 
Figure 2.5 Shared Situation Awareness ??????????????????52 
Figure 2.6: TSA, the goals and SA requirements??????...????????53 
Figure 2.7: Factors that build high levels of SSA and TSA ??????...????53 
Figure 2.8: Summary of literature review and theoretical framework ???????66 
Figure 3.1: Design Science Research Cycles ???????????.?????70 
Figure 3.2: General design cycle ????????????????...????74 
Figure 3.3: Dividing the problem in DSRM according to the thesis chapters ?....??76 
Figure 3.4: Steps of design ?????????????????...?????87 
Figure 3.5: System architecture of SAVER ???????????????..?82 
Figure 3.6: Evaluation method ?????????????????????84 
Figure 3.7: Overview of research methodology ???????????????85 
Figure 4.1: Steps of GDIA ?????????????????????...?90 
Figure 4.2: Steps of EGDIA ????????????????????...?91 
Figure 4.3: Ontology based situation awareness ?????????????.?97 
Figure 4.4: Example rules for detecting tsunami generation?????????....107 
Figure 4.5: Initial state of working space ???????????????.?107 
Figure 4.6: State of working space after rules scanning ??????????...?108 
Figure 4.7: Rule for tsunami generation ????????????????.?108 
Figure 4.8: SAVER system architecture for Situation assessment???????.....112 
Figure 4.9: OWL snippet showing data value assertion for earthquake magnitude........ 112 
Figure 4.10: OWL snippet showing rule for high intensity earthquake ?????...?113 
Figure 4.11: Fragment of ontology showing earthquake & tsunami parameters ?..?114 
Figure 4.12: Fragment of ontology showing various roles ????????..??115 
Figure 4.13: Fragment of ontology showing situation awareness representation ?.....115 
Figure 4.14: Fragment of ontology showing different phases of tsunami ???...?116 
11 
 
Figure 4.15: Situation ontology showing different concepts on abstract level ??116 
Figure 4.16: Object properties showing relationships between various concepts ??117 
Figure 4.17: Four different scenarios used in simulations ?????????...?118 
Figure 5.1: Personalised information for users using contextual information ???..132 
Figure 5.2: SAVER architecture design for personalised information ?????.?134 
Figure 5.3: Partial ontology using contextual concepts ???????????...136 
Figure 5.4: Partial ontology showing information requirements according to the various 
emergency phases and roles???????????????????....??136 
Figure 5.5: Interface showing levels of SA information about earthquake attributes 
??????????????????????????????...??139 
Figure 5.6: Interface showing three levels of SA information about wave information 
??????????????????????????????.?.......140 
Figure 5.7: Providing comprehension and projection about the evolving situation ?.?139 
Figure 5.8: Example SAGAT query ???????????????????.145 
Figure 5.9: Data for non-SAVER user ??????????????????.146 
Figure 5.10: Order of the experiments ??????????????????.147 
Figure 5.11: Order of experiments red circles showing the two groups of participants 
??????????????????????????????...??149 
Figure 5.12: Comparing performance of the SAVER users (orange circles) and non-
SAVER users (green circles) in the first session with the second session ???...?.?150 
Figure 5.13: Comparison of Individual SA measures with and without SAVER ??.151 
Figure 5.14: Comparison of the SA level 1 measures with and without SAVER ??.151 
Figure 5.15: Comparison of the SA Level 2 measures with and without SAVER ?....152 
Figure 5.16: Comparison of the SA Level 3 measures with and without SAVER ?....152 
Figure 5.17: Comparison of average time taken by a user to answer the queries and 
count of wrong answers using and without using SAVER ?????????.?154 
Figure 5.18: Comparison of SA performance based on experience using SAVER?.. 156 
Figure 5.19: Comparison of SA performance based on experience without SAVER...157 
Figure 5.20: HCI factors and their measurable attributes ??????????...159 
Figure 5.21: The example queries about the HCI of SAVER ????????.?159 
Figure 6.1: Overall goal and task division for the team of emergency managers ?.?168 
Figure 6.2: Co-ordination between NCMC, GEOC, and LEOC ??????.?172 
Figure 6.3: Example of Shared and Team Situation Awareness in GEOC ???..?173 
Figure 6.4: Team of teams involved in mass evacuation ??????????.?176 
12 
 
Figure 6.5: SAVER architecture to support SSA and TSA ????.??????179 
Figure 6.6: Partial example ontology for TSA and SSA support ???????.?180 
Figure 6.7: Explicit situation assessment provided by SAVER?????????181 
Figure 6.8: Example message from SAVER about team member SA ??????182 
Figure 6.9: Example message from SAVER asking for information ?????...?182 
Figure 6.10: Example message from SAVER about new information ??????183 
Figure 6.11: Example chat client for communication ?????????.???183 
Figure 6.12: Distribution of example questions and result analysis method ??.?...189 
Figure 6.13: Distribution of example questions and result analysis method ????190 
Figure 6.14: SSA measures with and without SAVER ??????????.?.191 
Figure 6.15: SSA level 1 measures with and without SAVER.?????????.191 
Figure 6.16: SSA Level 2 measures with and without SAVER ?????????192 
Figure 6.17: SSA Level 3 measures with and without SAVER ?????????192 
Figure 6.18: Team SSA score with and without SAVER ???.???????....192 
Figure 6.19: TSA score as common SA with and without SAVER ??????.?194 
Figure 6.20: TSA level 1 measures as common SA with and without SAVER ???194 
Figure 6.21: TSA level 2 measures as common SA with and without SAVER..............195 
Figure 6.22: TSA level 3 measures as common SA with and without SAVER ??. 195 
Figure 6.23: TSA score according to cognition and belief model ???..????..196 
Figure 6.24: TSA score as combined SA with and without SAVER ??????...197 
Figure 6.25: TSA Level 1 score as combined SA Level 1 with and without SAVER?198 
Figure 6.26: TSA Level 2 score as combined SA Level 2 with and without SAVER?198 
Figure 6.27: TSA Level 3 score as combined SA Level 3 with and without SAVER?199 
Figure 7.1: Various types and flows of information requirements ??????..?209 
Figure 7.2: Various ontologies in SAVER ???????????????..?211 
Figure 7.3: Architecture of operational SAVER?????????????. ?219 
Figure 7.4: SAVER on cloud ????????????????????..?222 
Figure 7.5: SAVER?s evolution to support SSA and TSA ?????????...?227 
 
 
 
13 
 
List of Tables 
Table 4.1: Planning and Intelligence (P&I) function information requirements??.?98 
Table 4.2: Operations function information requirements ??????????..100 
Table 4.3: Logistics function information requirements ?????????...?102 
Table 4.4: Welfare function information requirements ?????????.??..103 
Table 4.5: Controller function information requirements ????????.??..104 
Table 4.6: SAVER?s output from scenario 1 ??????????????.......119 
Table 4.7: SAVER?s output from scenario 2 ?????????????.??..120 
Table 4.8: SAVER?s output from scenario 3 ????????????..???.121 
Table 4.9: SAVER?s output from scenario 4 ??????????????.?..122 
Table 5.1: Boolean representation of various emergency phases ????????135 
Table 5.2: Distribution of participants ?????????????????.?144 
Table 5.3: A sample of SA query pool and example responses?????????148 
Table 5.4: Grouping of participants based on years of experience ????????.155 
Table 5.5: Responses to questions related to the usability of SAVER ?????.?160 
Table 5.6: Responses to questions related to the functionality of SAVER ????..161 
Table 5.7: Responses to related subjective measure of SA using SAVER ????...163 
Table 6.1: SSA level 1 requirement of Situation Reporting Section (P&I)?????175 
Table 6.2: SSA level 3 requirement of Situation Reporting Section (P&I)?????177 
Table 6.3: TSA level 2 requirement of Situation Reporting Section (P&I)????...177 
Table 6.4: Example SSA queries ????????????????????186 
Table 6.5: Example TSA queries ????????????????????187 
Table 6.6: Distribution of participants in teams ???????????????188 
Table 7.1: Levels of SSA and TSA ??????????????????.?207 
Table 7.2: Comparison of experienced and novice emergency managers? performance 
with and without SAVER ??????????????????????.213 
Table 7.3: Correlations between of TSA, common SA and TSA as cognition and belief 
model ???????????????????????????.?. ?217 
 
 
 
14 
 
List of Publications 
Javed, Y., Norris, A. C., & Johnston, D. M. (2012). Situation Awareness and the Design 
of Emergnecy Management Systems for Mass Evacuation: A Literature Review.  
(Under review). Elsevier: Decision Support System and Electronic Commnerce. 
Javed, Y., Norris, A. C., & Johnston, D. M. (2012). Improving Situation Awareness for 
Mass Evacuation Decision Making Protype SAVER (Under review). InderScience: 
Special Issue in An Interdisciplinary Apporach to Crisis Response and Management. 
Javed, Y., & Norris, A. C. (2012). Design, Implementation and Evaluation of an Emergency 
Management System to enhance Situation Awareness. General Sir John Kotewala 
Defence Univeresity Annual Symposium 2012, Ratmalana, Srilanka.  
 Javed, Y., Norris, A. C., & Johnston, D. M. (2012). Evaluating SAVER: Measuring Shared 
and Team Situation Awareness of Emergency decision makers. In Proceesings of 
9th International Conference on Information Systems for Crisis Response and 
Management (ISCRAM 2012), Vancouver, Canada, pp. 1-11.  
Javed, Y., Norris, A. C., & Johnston, D. M. (2011). Mass Evacuation Support System 
(MESS). Validating Information Requirements of Emergency Managers. In Proceedings 
of 7th International Conference on Information Technology and Applications 
(ICITA 2011). Sydney, Australia. 
Javed, Y., Norris, A. C., & Johnston, D. M. (2011). Combining Naturalistic and Rational 
Decision Models for Designing Mass Evacuation Support System (MESS). In Proceedings 
of Intellect International Consortium Academic Conference, 2011. Sydney, 
Australia. 
Javed, Y., Norris, A. C., & Johnston, D. M. (2011). Information Requirements for Designing 
Computerized Collaboration System. In Proceedings of Institute of Information and 
Mathematical Sciences (IIMS) 7th Postgraduate Student Conference,Massey 
University, Auckland, New Zealand. 
Javed, Y., & Norris, A. C. (2011). Prototyping and Evaluation of a Computerised Emergency 
Management System Based on an Ontological Inference Design. In the Proceedings of 
15 
 
IEEE Internal Global Humanitarian Conference( GHTC), Seattle, Washington, 
USA, pp. 462-466.  
Javed, Y., Norris, A. C., Johnston, D. M., & Doyle, E. H. (2011) Towards a Framework 
for Crisis Decision Support Systems: Information Requirements for Contextual 
Team Situation Awareness In: CutterIT Journal: Special Issue on IT and Crisis 
Management, 24(1). 26-33 
Javed, Y., Norris, A. C., & Johnston, D.M. (2011). Ontology based Inference to Support Team 
Situation Awareness in Emergency Management. In Proceedings of the Information 
Systems for Crisis Response and Management (ISCRAM 2011), Lisbon, 
Portugal. 
Javed, Y., Norris, A. C., & Johnston, D. M. (2011). Using Ontology to Enhance Context based 
Team Situation Awareness. In Proceedings of New Zealand Computer Science 
Research Student Conference (NZCSRSC 2011), Palmerston North, New 
Zealand. 
Javed. Y., Norris. A. C., Johnston. D.M. (2010). A Design Approach to an Emergency 
Decision Support System for Mass Evacuation. In Proceedings of 7th International 
Conference on Information Systems for Crisis Response and Management 
(ISCRAM). Seattle, Washington, USA. 
Javed. Y., Norris. T., Johnston. D.M., (2009). Design, Implementation and Evaluation of Web-
based Incident Information Resource Management System (WIIRMS) for decision support in 
Emergency Operation Centres. In Proceedings of International Conference of 
Enterprise Information Systems and Web Technologies (2009), Orlando, 
Florida. USA. 
Javed. Y, Norris. A. C., & Stockdale. R. (2008). Key IT-based Capabilities for Disaster 
Management and Problems in implementing related promising technologies(Short paper & 
poster). In Proceedings of 2nd Australasian Disaster Management Conference 
2008. Wellington , New Zealand. 
 
 
16 
 
 
INTRODUCTION 
?The best way to predict the future is to invent it? 
(Alan Curtis Kay) 
 
1.1 Aim of the Research 
This research investigates the decision making process of emergency managers in 
various Group Emergency Operations Centres (GEOCs) of New Zealand. GEOC has 
to integrate information from multiple local Emergency Operation Centres (EOCs) and 
many other organisations such as the Fire Service, Police, Welfare organisations etc. to 
develop a team situation awareness (TSA), which is required for decision making. 
GEOCs also have to co-ordinate the implementation of decisions in the form of actions 
and tasks by various emergency management teams that require a shared situation 
awareness (SSA). This research examines how emergency managers develop their 
individual, shared and team Situation Awareness (SA) in GEOCs.  
Many emergency response failures are due to the inappropriate or delayed 
understanding of the emergency situation at an individual or team level. Examples 
include deaths during the Indian Ocean tsunami in 2005 due to the late or no 
evacuation decision, casualties in the Japan tsunami in 2011 due to the delayed 
evacuation decision, three mile nuclear incident (Janex, 2007), fire fighters death in 9/11 
(Son et al., 2009), the mass casualty during the Bhopal incident (Endsley, 1999). In all of 
these cases, decision makers failed to comprehend the actual situation or to implement 
the decisions in time.   
Much of the available research has been done on an individual?s SA and how it can be 
supported and measured but far less has been published on SSA and TSA, which are 
more important for large events in which teams of people are involved in decision 
making and implementation. Individuals from different backgrounds, cultures, 
experiences and, most importantly, with different goals in mind see situations 
differently. However, it is very important to reach an accurate and common 
understanding of the situation as rapidly as possible so that correct decisions can be 
17 
 
made quickly. Similarly, hundreds of different organisations may be involved in the 
implementation of decisions in the form of actions, co-ordination between the 
individual members, teams and their activities; it is vital to reduce avoidable repetition, 
particularly as the situation may be fluid and changing frequently. Emergency situations 
can clearly become very complex. 
The study described in this thesis draws upon the experiences derived from simulated 
exercises, namely Ruaumoko1 and Tangaroa2. Moreover, expert opinions obtained by 
interviews and questionnaires are used to understand the information requirements and 
flows required for SA development and decision making. This knowledge is then used 
to design a decision support system dubbed Situation Aware Vigilant Emergency 
Reasoner (SAVER) to support SA, SSA, and TSA of emergency managers to improve 
overall decision making during large-scale emergency events. A series of simulated 
experiments were performed to validate the findings and evaluate whether SAVER can 
really improve SA, SSA and TSA or not by measuring the SA, SSA and TSA of the 
participants.  
The following section describes the research contribution of this study, and the next 
section contains an overview of the emergency management research. The subsequent 
sections describe the research context, problem, objectives, and the scope of the inquiry 
and finally, an overview of the whole thesis.  
1.2 Research Contributions 
1. The identification of information required for developing SA, SSA, and TSA 
during mass evacuation operations using a cognitive tasks analysis technique 
2. The recognition of the need of SSA and TSA during mass evacuation operations 
                                                 
 
1 Ruaumoko is the name of the Maori mythological God who is guardian of earthquakes and volcanoes. 
The aim of Exercise Ruaumoko was to "test New Zealand's all-of-nation arrangements for responding to 
a major disaster resulting from an impending volcanic eruption in Auckland." 
 
2 The name for the exercise comes from Maori mythology where ?Tangaroa? is the God of the sea, rivers, 
lakes and all that live within them. Exercise Tangaroa was a national, multi-agency exercise that was held 
on 20 October, 2010. The exercise was led by the Ministry of Civil Defence & Emergency Management 
and was supported by the 16 CDEM Groups, central government departments, emergency services, 
lifeline utilities, and other agencies as appropriate. 
 
18 
 
3. The classification of the information requirements according to the mass 
evacuation operation phases and the emergency management roles   
4. The use of ontologies for information classification according to the SA 
requirements and the SA levels in the computer based system 
5. The development of ontologies for the information requirements of SA, SSA 
and TSA, the emergency management roles, and the emergency situation 
involving a mass evacuation 
6. The design of ontology based context architecture for personalised information 
sharing to reduce the information overload  
7. The design and development of the computer based system prototype (SAVER) 
for improving individual, shared and team SA of emergency managers  
8. The design and development of a computer based system (SAVER) improving 
the TSA of emergency managers according to all three prevalent models of TSA 
9. The evaluation of SAVER using SAGAT technique proving that use of SA 
oriented system design can improve individual SA using the computer based 
system 
10. The extension of SAGAT for measuring SSA and TSA of emergency managers 
11. The evaluation of SAVER using three different TSA models 
12. The formalisation of SA, SSA and TSA definitions 
13. The division of SSA and TSA into three levels for detailed modelling  
14. The discovery of relationships between three different models of TSA  
1.3 Theoretical Framework 
1.3.1 Emergency Management overview  
With the recent increase in serious disasters worldwide, the importance of improving 
emergency management practices has greatly increased. According to the Merriam 
Webster dictionary, an emergency is an unforeseen combination of circumstances or the resulting 
state that calls for immediate action (Merriam-Webster, 2011). Common examples are floods, 
earthquakes, tsunamis, volcanic eruptions, hurricanes, tornados, or acts of terrorism etc. 
that can bring damage or loss. Nowadays, disasters are seen as the result of 
inappropriately managed risk, which is the product of hazards and vulnerability. Hazards 
in the areas with low vulnerability are not considered as disaster, as is the case in 
unoccupied regions (Quarantelli, 1998).  
19 
 
Emergency management is an inter-disciplinary field dealing with the strategic  
organisational processes used to shield critical assets (living or non-living like 
infrastructure or economy) from hazards that can cause disasters or catastrophes, and to 
ensure the continuance of the organisation within their planned lifetime (Haddow, et al., 
2007). Emergency management is divided into four phases namely Mitigation, 
Preparedness, Response and Recovery (Quarantelli, 1998) as shown in Figure 1 below. 
Mitigation deals with the steps taken to reduce the risk and to prevent hazards from 
evolving into disaster or to minimize the impact of disaster. Mitigation actions can be 
structural or non-structural. Structural actions use technological solutions like flood 
levees, earthquake resistant buildings and bridge structures etc. Non-structural measures 
include legislation, land-use planning, insurance etc. (Wilson, 2009). Preparedness is a 
cycle of planning, organising, training, equipping, exercising, evaluation, and 
improvement activities to ensure the effective co-ordination and the enhancement of 
capabilities to prevent and respond to the effects of natural and man-made disasters 
(FEMA, 2007). In the preparedness phase, plans of the actions are developed to manage 
and counter the risks and take actions to build the necessary capabilities needed to 
implement such plans. The Response phase includes the mobilisation of the core 
emergency services and first responders such as fire fighters, police, and ambulance 
crews etc. in the disaster area. Though this phase requires different actions depending 
upon the type of disaster, large events such as a tsunami or volcanic risk require the 
evacuation of people before the incident, as well as rescue and relief during or after the 
event (FEMA, 2007). The recovery phase starts after the immediate threat to human life 
has ended. During this phase, actions are taken to bring the community back to normal. 
Recovery activities continue until all systems return to the normal or better. Recovery 
activities include returning vital life-support systems to minimum operating standards, 
e.g. temporary housing, grants, medical care etc.  
 
Figure 1.1 Emergency management cycle 
Mitigation 
Preparedness 
Response 
Recovery 
20 
 
Although decisions are made during every phase of emergency management, decision-
making becomes especially important during the response phase. In the case of crises 
like a tsunami or volcanic eruptions, decisions have to be made about the confirmation 
of the threat, possible area under threat, time of occurrence, areas to evacuate, 
evacuation start time etc. During decision making, multiple emergency managers from 
various organisations interact with each other and with the domain experts to 
understand the situation before making a decision and carrying out their response tasks. 
Incoming information from various sources is usually general and not necessarily 
intended or sufficient for a specific decision, role, or task. Consequently, information is 
processed and understood using an individual?s perception which can widely vary 
according to their background, culture, experiences, stress level or attention level at that 
time. Hence, the individual?s understanding, which feeds into the collective 
understanding of the team involved in the decision-making, may not be accurate.   
1.3.2 Decision Making in Emergency Management  
Emergency decision or ordinary decision making consists of two parts: first, 
understanding the situation by processing the situational information cues and second, 
suggesting actions to normalise the situation or to make it safe. Incoming information is 
input to the whole process. Decision making during an emergency situation is different 
from ordinary decision making in many ways e.g.  
 Stress level is very high as lives and structures are at stake 
 Limited time is available to make decisions 
 The situation is continuously changing  
 Incoming information is uncertain and can quickly become out of date 
 Information can be either incomplete, or in some cases excessive, leading to 
overload and the need to extract relevant from general information  
These factors affect the information processing by individuals and teams as they try to 
understand the situation. As information from multiple sources and domains has to be 
integrated into a meaningful and usable format it becomes very time consuming to 
classify it according to the requirements of different people who have different roles and 
goals.  
21 
 
Though disasters like tsunamis cannot be prevented, their impact on society can be 
minimised by good decision making and by making use of the disciplines such as 
engineering, management, policy making and community support, which are all related 
to emergency management.  According to Hough (2008), in the last decade more than 
one million people have died and multibillion-dollar losses incurred due to the direct 
impact of natural disasters like tsunamis, floods and earthquakes ? this figure does not 
include causalities from the recent Japan earthquake and tsunami of 2011. During the 
2005 Indian and 2011 Japan tsunamis, ineffective use of the incoming information 
needed for timely evacuation led to mass destruction and casualties.  Hence, right and 
timely decision making is the most important part of emergency management as it 
defines the actions required to normalise the situation or arrange safe evacuation if 
required. Information technology (IT) is playing a major role in providing support to 
decision makers in the form of decision support and emergency management systems. 
However, most of these systems are general purpose systems and the information 
provided by them needs further processing before use, which adds to decision delays.   
1.3.3 Importance of SA/TSA and SSA in Decision Making 
The process of understanding the prevailing situation is called situation assessment, this 
process results in a product named situation awareness (SA). SA is the degree to which a 
person understands the situation. Several studies have described SA as a crucial factor 
for better decision making (Bryant, 2002; Klein, 2000) and the seminal work by Endsley 
and others has shown its close relationship with emergency decision making during time 
critical and complex situations (Adam et al., 1995; Endsley, 1995a; Smith & Hancock, 
1995). As indicated, the exchange of individual Situation Awareness (SA) between 
managers leads to team situation awareness (TSA) and more accurate and rapid decision 
making. The sooner team members understand the full picture of a situation, the sooner 
they can develop consensus and suggest actions to normalise it. Once a decision is 
made, it has to be implemented by emergency managers/officials from multiple 
organizations in a co-ordinated way.  For this, they have to know about their own 
responsibilities in the overall operation and their role in relation to others? activities. 
This understanding is called shared situation awareness (SSA). Hence, TSA is important 
for the team decision making, whereas SSA is important to the implementation of the 
decision or to any collaborative task discharged in a co-ordinated way. In the absence of 
SSA, every team member would be concentrating on only his/her part, ignoring his or 
22 
 
her role in the team. This not only results in the repetition of various activities but it can 
also delay or curtail tasks leading potentially to the failure of the whole mission due to 
the inability of team members to integrate their individual and shared awareness.  
Some recent research to improve SA in emergency decision making has made use of 
computer based systems, e.g. (Bergstrand & Landgren, 2009; Betts et al., 2005; 
Lanfranchi & Ireson, 2009; Prasanna et al., 2009). Several studies (O?Connor et al., 2008; 
Endsley & Robertson, 2000; Endsley & Rodgers, 1994; Resnick, 2003; Riley et al., 2006) 
have also explained how a system fulfilling SA requirements can maintain the required 
level of SA and better support such decision making.  However, the studies noted above 
underestimate the critical SSA and TSA factors. At the same time, very little work has 
been done to see how SA, SSA or TSA can be developed for decision making and 
implementation during mass evacuation in case of a tsunami risk.  
Another important issue related to the design of computer systems supporting SA, SSA 
and TSA is the gathering of information requirements. The identification of information 
required to develop and maintain necessary degrees of SA, SSA and TSA is a challenge 
since, apart from physical tasks, many cognitive processes are also involved in situation 
assessment. Therefore, innovative techniques are required to gather the information 
requirements of physical tasks as well as cognitive processes of emergency managers 
during team decision making and collaboration.  
This thesis aims at understanding the SA, SSA, and TSA requirements of emergency 
managers for decision making and collaboration during mass evacuation operations. 
Moreover, based on the requirements, it describes the design, development, and 
evaluation of a prototype computer system (SAVER) to improve SA, SSA and TSA.  
1.3.4 Ontology based SA 
Ontology originated from the area of philosophy (Gruber, 1993).  In philosophy, 
ontology is the science of being or existence.  In Information Technology (IT) and 
Computer Science, however, ontologies are used to define relationships between various 
concepts to conceptualise the domain or area. More specifically, ontologies can be used 
as a language for describing content and its structure and the properties and 
relationships between concepts in a structural and consistent manner. Ontology is 
defined as a formal, explicit specification of a shared conceptualisation (Studer et al., 
1998). The ?conceptualisation? refers to an abstract model of some phenomenon in the 
23 
 
world by identifying the relevant concepts of that phenomenon. The ?explicit? means 
that the type of concepts used, and the constraints on their user are explicitly defined.  
The term ?Formal? refers to the fact that the ontology should be in a standard language, 
which is machine-readable. ?Shared? reflects the notion that ontology captures a 
consensus, that is, not the private idea of some individual, but the accepted and shared 
understanding of a group.  
Ontologies describe objects using individuals (instances), classes (concepts), attributes, 
and relationships (properties). Classes are the sets of objects (individuals) that share the 
same characteristics. Individuals include real objects, for example, humans, animals, 
cars, etc. Attributes are the characteristics of classes and relationships are the 
associations between classes.  From the above explanation, it can be seen that an 
ontology can be used to model situation concepts, a procedure necessary for computer 
analysis of a situation. At the same time, the ontology should represent the shared 
understanding of the concepts leading to the shared understanding of a situation by a 
group of people with different backgrounds, roles and objectives. An ontological 
description can therefore facilitate TSA. Another important aspect of ontology use is its 
specification in a computer-readable language making ontology ideal for computer 
based situation analysis. Several studies (Boury-Brisset, 2003; Horney et al., 2003; 
Mathues et al., 2003; McGuinness et al., 2003) have proposed ontology based SA 
systems. They also provide higher level situation ontologies to capture generic situation 
descriptions that can be contextualized to derive more domain specific ontologies.  
The approach described here in this thesis, endeavours to extend the existing SA 
ontology for SSA and TSA by modelling contextual and situational information. In 
situations like volcanic eruptions, floods, tsunamis etc., when expert opinion is required 
to decide, for example, when to evacuate, which area to evacuate first etc., it is often 
difficult for emergency managers to quickly integrate information from diverse sources 
and make sense of it. Moreover, as multiple organisations are involved at different levels 
of decision making and collaborative tasks, it is impractical to memorise the information 
requirements of all fellow team members or members from other organisations who 
depend on them and vice versa. In these circumstances, the necessary levels of SSA and 
TSA are not achieved. Ontology can be used to model the situation and context in a 
detail to allow automatic integration of information from diverse sources, provide 
semantics to the situational objects and most importantly, allows inference and 
24 
 
reasoning capabilities necessary to develop judgements about the understanding of a 
situation. In this way, they can see how the current situation affects their objectives in 
relation to other members of the team. This reduces the chances of misunderstanding 
and the emergency managers can fully understand the dynamics of the evolving 
situation and its implications. Hence, this study also investigates the possible application 
of ontologies in the computer based system supporting SA, SSA and TSA.   
1.4 Research Context 
In New Zealand, the National Crisis Management Centre (NCMC) looks after a crisis at 
national level and co-ordinates with the regional emergency offices known as Group 
Emergency Operation Centres (GEOCs). GEOCs in turn, co-ordinate with local 
Emergency Operation Centres (LEOCs), within that region. Co-ordination between 
NCMC and GEOCs and LEOCs is critical for SA and decision making as most of the 
major decisions are made by NCMC and implemented by LEOCs and supporting 
organisations such as the Fire Service, Police etc. 
A system known as the Co-ordinated Incident Management System (CIMS3) is used for 
managing the response to an incident involving multiple responding agencies. CIMS has 
five basic functions: Controller, Planning & Intelligence (P&I), Operations, Logistics 
and Welfare. All these functions have their own responsibilities during various phases of 
an emergency situation and need to share situation and co-ordination information. 
Hence, they are dependent on each other for decision making and collaborative tasks. 
This research aims at identifying and supporting the requirements of SA, SSA and TSA 
needed for the effective decision making and collaboration.  
To explain the approach, we consider a tsunami scenario. New Zealand has a coastline 
of 15,134 km and resides in the South-West Pacific Ocean, which is the world?s most 
dangerous area for highly destructive tsunamis. First, individual/team members use 
seismic and related data to register an underwater event (e.g. earthquake or volcanic 
explosion) that could cause a tsunami. Deep-Ocean Assessment and Reporting Tsunami 
(DART) data are then analysed to confirm whether a tsunami has been generated or 
not. Once tsunami generation is confirmed, areas of possible devastation are marked 
                                                 
 
3 http://en.wikipedia.org/wiki/Coordinated_Incident_Management_System 
25 
 
and planning for possible evacuation is started. Although the priority zones are 
designated, decision makers must remain aware of other at risk areas. Similarly, 
decisions to evacuate, and the time and extent of evacuation, should take into account 
schools, hospitals, and other buildings that may need special attention.  
Planning for the safe evacuation of people includes transport, food and shelter, and 
medical facilities are critical considerations since wrong decisions can lead to either 
unnecessary disruption or increased casualties and economic loss. The safe return of 
evacuees, once water levels have returned to normal, is also a major factor dependent on 
damage, access and services. While evacuees are away from their properties, proper 
security arrangements are also important. These planning steps have many 
considerations for individual and team situation awareness. The present research is 
undertaken to identify and support the requirements of SA, SSA and TSA of the various 
emergency management roles to improve their decision making and task collaboration 
during the tsunami threat.  
1.5 Research Objectives and Questions 
As this research is the first of its kind in this domain and geographical area, some very 
fundamental questions need to be answered. The main research objective is to find a 
way to improve SA, SSA and TSA using a computer based system. The following 
research questions are framed to achieve this objective.  
1 How to design a computer based system for automatic situation assessment? 
2 How to design a computer based system supporting human SA in mass evacuation 
operations? 
3 How to design a computer based system supporting shared and team SA (SSA and 
TSA) of emergency managers in mass evacuation operations? 
1.6 Research Approach  
Design Science Research Methodology is used to answer the questions mentioned 
above. Design Science (DS) is a problem solving paradigm whose goal is to produce an 
artefact by using the analysis, design, implementation and evaluation of the artefact. 
Working with the technology, going through the design and development of the artefact 
and understanding the issues with the usage of artefact is at the core of this research 
paradigm. The knowledge generated by this research informs us how an artefact can be 
improved and how it is better than the existing solutions. 
26 
 
DSR in information systems provides a solution to what are considered to be the 
?wicked problems? (Brooks 1987; Rittel & Webber 1984). Cases with such problems 
have the following characteristics: 
 Ill-defined environments where user requirements are unclear and unstable 
 Complex problems with many intermingling sub-problems 
 Design processes flexible to changes 
 Effective solution depends upon human cognitive abilities e.g. intelligence or 
creativity 
 Effective solution depends upon human social abilities (e.g., teamwork) 
All of the above problems are evident in the design of information systems to be used 
for decision making and planning in emergencies. Therefore, this study uses Design 
Science Research Methodology proposed by Peffers et al. (2008). Peffers et al. provide a 
commonly accepted framework for carrying out successful design science research in 
information systems and avoids more ad hoc approaches. In addition, to capture the 
comprehensive information requirements of cognitive processes along with the physical 
tasks, a cognitive task analysis technique called Goal Directed Information Analysis 
(GDIA) (Prasanna et al., 2009) is used. The evaluation of the design and prototype 
(artefact) is done by measuring the SA, SSA and TSA of emergency managers using 
Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 1995b) in 
simulation experiments.  
1.7 Scope of the Research Enquiry  
It is very important to clearly identify and describe the scope of the study. The scope of 
this study is described as follows:  
 In this study, research explorations are limited to only Group Emergency 
Operation Centres (GEOCs) instead of all the organisations involved as it is 
very difficult for an in-depth study in the due period.  
 Field observations are confined to the national level exercises and simulations 
i.e., Ruaumoko and Tangaroa or any other evaluation experiments done using 
similar scenarios.  
 The research is focused on emergency decision making during mass evacuation 
in case of tsunami risk. 
27 
 
 Investigation is carried out only for the most important roles of GEOC, which 
are Controller, Planning and Intelligence, Operations, Logistics, and Welfare 
functions who work as team members.  
 This research is focused on the management and co-ordination required for 
decision making and implementation instead of the scientific and technical 
details of tsunami scenarios. Thus, this research will not provide the scientific 
interpretation of sensor data etc. All the data values used will be simulated but 
realistic.  
 The prototype is developed for specific simulated scenarios and as a proof of 
concept to evaluate the model and validate the findings.  
 The interfaces are not developed to fulfil the information requirements of all the 
roles; only those interfaces are developed which are needed in evaluation for the 
proof of concept.  
1.8 Overview of Thesis 
This section describes the overall organisation of the thesis and explains the chapter 
construction. 
1.8.1 Chapter 2: Theoretical Foundation and Literature Review 
The aim of this chapter is to develop a theoretical foundation of the study and highlight 
the gap in literature related to the use of information systems for improving SA, SSA 
and TSA in emergency decision making and implementation. The structure of this 
chapter is: 
 Emergency management and emergency decision making 
 Theoretical underpinning of SA, SSA and TSA  
 Information systems to improve SA 
 Ontology based SA 
 Existing ontology based systems 
The chapter identifies the significant gap in the literature concerning the use of 
information systems to support SSA, and TSA in decision making and implementation 
during emergencies involving mass evacuation.  Another gap found in the literature is 
about the information requirements of SA, SSA and TSA, for various emergency 
management roles. 
28 
 
1.8.2 Chapter 3: Research Methodology 
The aim of this chapter is to provide an overall design of the research methodology 
used to achieve the research objectives. The main aspects are as follows: 
 The overall design research architecture and theoretical foundations for its 
suitability to this study 
 The use of a modified form of Goal Directed Information Analysis (GDIA) 
(Prasanna et al., 2009) to obtain the information needs of various emergency 
management roles 
 The design and development of the prototype system for improving SA, SSA 
and TSA 
 The use of ontologies in prototype development for automatic situation 
assessment 
 The Evaluation of the prototype to evaluate the model and validate the findings 
 The division of the forthcoming chapters according to three inter-linked 
problems  
The main outcome of this chapter is the overview of the design research methodology 
and clear direction for research activities to achieve the research objectives of the study. 
Other outcomes are the steps for design, implementation and evaluation of a prototype 
that can support SA, SSA and TSA. Moreover, the main research problem is divided 
into three sub problems and is tackled in three separate chapters i.e. 4, 5 and 6 discussed 
below.   
1.8.3 Chapter 4: SAVER: Situation Aware Vigilant Emergency Reasoner   
This chapter discusses the problem of gathering requirements needed to develop 
situation awareness (SA) and the design of a system that can model situations and 
provide inference to classify incoming information into information required to develop 
three levels of SA. The chapter then suggests a solution to the problems followed by the 
design and development of a prototype. In the next section, this chapter describes the 
evaluation of the prototype designed and developed. The last section discusses the 
conclusions. The chapter is structured as follows: 
 The problem identification 
 The suggested solution to the problem 
29 
 
 The design and development started with information requirements gathering.  
 The selection of scenarios used in national exercises 
 The description of interview processes and observations during the exercises 
 Data validation via questionnaire from GEOCs all over New Zealand 
 Prototype development  
 Testing of the prototype 
 Conclusions 
The chapter outcomes are the process of SA development and the information 
requirements of SA. Other outcomes include a prototype designed and developed using 
ontologies for situation modelling and the testing results of the prototype evaluation. 
The prototype uses ontology based inference to classify situation information according 
to the SA requirements. 
1.8.4 Chapter 5: Improving Individual Situation Awareness Using SAVER 
The aim of this chapter is to describe the problems of developing accurate situation 
awareness (SA) necessary for decision making and carrying out collaborative tasks. 
Moreover, in the next section, personalised and contextualised information is suggested 
as a solution. Following the suggestion section, design and development of the 
prototype is conversed in detail. This section proposes ontology based contextual SA 
information to provide personalised information. The chapter then provides the details 
of simulated experiments conducted to measure the SA of emergency managers to 
evaluate the prototype design. This chapter is structured as: 
 The problems related to individual SA 
 The suggested solution 
 The design and development of the contextual SA component of the prototype 
 The development of various ontologies 
 The description of features of prototype interfaces to improve SA 
 The evaluation of the prototype by emergency managers in experiments using 
SA measurement techniques 
Outcomes include the architecture design of a contextual SA information broker, 
developed ontologies and a prototype system with SA oriented user interface design for 
enhanced SA evaluation results of the prototype.  
30 
 
1.8.5 Chapter 6: Improving Shared and Team Situation Awareness Using SAVER  
Chapter 6 starts with the description of the problem relating to shared and team 
situation awareness (SSA & TSA). It describes the requirement of developing and 
maintaining the accurate SSA and TSA necessary for team decision making and 
collaborative tasks. In the next section, a solution to the problem is suggested. 
Afterwards, the design and development of a prototype to support SSA and TSA is 
described. This section also explains the role of ontologies in the development of a 
system supporting SSA and TSA. In the next section the prototype is evaluated by 
measuring SSA and TSA of emergency managers in experiments and the results are 
reported. Conclusion is the last section of this chapter. This chapter is structured as: 
 The problems related to SSA and TSA 
 The solution suggested to the problem 
 The design and development of the prototype component to support SSA and 
TSA 
 The use of ontologies in the prototype development 
 The description of the prototype features to improve SSA and TSA of 
emergency managers 
 The results of the prototype evaluation in experiments 
Outcomes of this chapter include the architecture design of an ontology based SSA and 
TSA information broker, ontologies developed to support SSA and TSA, a prototype 
system, evaluation results of a prototype for supporting SSA and TSA.  
1.8.6 Chapter 7: Discussion  
Chapter 7 provides the formalisation of SA, SSA and TSA definition on the basis of 
results. In addition, it discusses the implications of the results in relation to the existing 
theories and the limitations of the study. The chapter also elaborates on the possible 
relationship between different models of TSA and their relation with the individual SA. 
In the next section of the chapter, implementation requirements of operational SAVER 
are discussed. At the end of the chapter, a summary of the whole study is provided.   
1.8.7 Chapter 8: Summary, Conclusions and Future Work 
The final chapter outlines the conclusions in the form of answers to the research 
questions and points to their details in the thesis. Moreover, in the next and last section 
31 
 
of the chapter, possible avenues of future research related to the study are discussed. See 
Figure 1.2 below for the overall flow of the thesis.   
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 1.2 Flow of thesis chapters 
 
 
 
 
 
  
Chapter 1  
Introduction 
Chapter 2  
Literature Review 
Chapter 3      
 Design Science 
Research Methodology  
Chapter 4  
SAVER 
Chapter 5  
SAVER: SA 
Chapter 5  
SAVER: SSA & TSA 
Chapter 7  
Discussion 
Chapter 8  
Conclusions & Future 
Work 
32 
 
 
THEORETICAL FOUNDATION AND 
LITERATURE REVIEW 
2.1 Introduction 
The significant increase in the intensity and frequency of natural disasters such as 
tsunamis, earthquakes, floods and hurricanes has led to an increase in the number of 
computer systems supporting the co-ordination, situation awareness and overall 
decision making of emergency managers. Some recent events such as the 2004 Indian 
Ocean tsunami, when more than 288,000 people died in a dozen countries (Said, 
Ahmadun, Mahmud, & Abas, 2011), Hurricane Katrina, responsible for 1836 causalities 
(Knnab, Rhome, & Brown, 2011), and the September 2009 American Samoa tsunami 
that caused around 2000 causalities (Satake, Rabinovich, Kanoglu, & Tinti, 2011), all 
displayed a failure of co-ordination and management which resulted in delayed 
evacuations and poor emergency responses.  
In these complex and dynamic situations, the high volume of diverse information from 
multiple sources can readily lead to overload and make it difficult for emergency 
managers to relate the information to the prevailing situation. In addition, the raw 
information may need further transformation to make it applicable to the different roles, 
objectives and tasks of the emergency managers ? an unwelcome overhead when time is 
of the essence and stress levels are high. In addition, the different managers in the 
emergency centre(s) must share information and co-operate in their decision making to 
achieve overall as well as local objectives. 
This chapter contains a complete literature review on emergency management, 
emergency decision making, and information systems support in emergency decision 
making and related concepts. It focuses on systems for mass evacuation, the central role 
of situation awareness and the contribution of ontologies to systems design.  On the 
one hand, this chapter describes the basic knowledge necessary to understand the 
purpose of this thesis, and on the other hand, it discusses some similar research studies 
to highlight the missing literature and establish how this work is different.  
33 
 
2.2 Emergency Management 
An emergency is defined as an abrupt and unforeseen event endangering the safety of 
people, property or the environment that compels instant action (Cronan, 1998). An 
emergency is also a threat to the basic structures or the fundamental values of a social 
system, which requires making critical decisions under time pressure and highly 
uncertain circumstances (Rosenthal et al., 2001). These definitions apply to all types of 
emergency situations. Emergency management is defined as the planning for, and the 
execution of, all emergency functions necessary to mitigate, prepare for, respond to and 
recover from emergencies and disasters (FEMA, 2007).  
Emergency management deals with risk mitigation, response and recovery. It is also 
defined as ?a range of measures to manage risks to communities and the environment? 
(AEMI, 2008). Some emergency management frameworks are as follows:  
New Zealand: The National Civil Defence Emergency Management Planning has the 
4R approach to EM comprising of Reduction, Readiness, Response and Recovery 
(MCDEM, 2009). 
Australia: The national government considers EM as Emergency Risk Management, 
Disaster Mitigation and Consequence Management (AEMI, 2008). 
United States: The Federal Emergency Management Agency (FEMA) has defined a very 
detailed EM strategy for both public and business. In general, it ranges from mitigation 
to response and recovery (FEMA, 2007). 
Canada: Canadian EM framework covers Prevention and Mitigation, Preparedness, 
Response and Recovery (PSC, 2011). 
Almost all of these definitions introduce four functional phases namely; Mitigation, 
Preparedness, Response and Recovery in emergency management. All these functions 
are discussed in more detail in the upcoming sections. The complexity and importance 
of emergency management activities have been illustrated in recent disasters such as the 
1995 Kobe Earthquake, the 2004 Sumatra Earthquake and tsunami, the 2005 Pakistan 
Earthquake, the 2005 Hurricane Katrina, the 2010 Haitian Earthquake and the 2011 
Japanese Earthquake and tsunami. All these disasters bring to light the vast range of 
situations to be considered while planning emergency management and the constraints 
in existing practices, procedures and systems.   
34 
 
2.2.1 Emergency management in the context of mass evacuation 
The mass evacuation of an area is needed when a natural or technological hazard 
threatens and risks the safety of those within the area. Mass evacuations can also be 
needed following the impact of a hazard which has subsequently made the area 
unliveable. Evacuation becomes necessary when the benefits of leaving outweigh the 
risk of sheltering-in-place (MCDEM, 2008a). There are two main options available to 
emergency managers to direct people, when considering ordering an evacuation; 1) 
shelter-in-place, and 2) evacuate. These two options can be used separately or in 
combination to provide the most effective response to the circumstances of a specific 
event (MCDEM, 2008a).  
Shelter-in-place: This option requires people to remain in their current location, e.g. 
home, office etc. Shelter-in-place is necessary when the nature of a hazard shows that 
leaving would place people at greater risk. It also carries the benefit that people generally 
recover from an emergency situation faster in familiar areas. Dangers that may need 
shelter-in-place include (MCDEM, 2008a):  
 Toxic chemical spill 
 Radiation release 
 Act of terrorism  
 Pandemic  
 Volcanic ash fall 
Evacuation: Evacuation becomes necessary when the benefits of leaving outweigh the 
risk of staying. This option means asking or directing people to leave their current, 
potentially dangerous location and providing help to them in the form of transport, 
shelter and the other necessities of life. Evacuees should be kept as near to their homes 
as possible. Hazards that may require evacuation include (MCDEM, 2008a): 
 Tsunami/hurricane/tornado  
 Flash flooding 
 Riverine flooding  
 Volcanic activity (other than light ash fall)  
 Landslip/landslide 
 Wildfire 
35 
 
 Lifeline utility/infrastructure failure 
Evacuation may take place before or after an event. Evacuation can be further classified 
into two types: voluntary evacuation, or mandatory evacuation. Mandatory evacuation is 
instructed when risk to the residents is imminent and it is highly dangerous to remain 
where they are, whereas voluntary evacuees leave their current location because of actual 
or perceived risk without being directed to do so. Volunteer evacuation is beneficial in a 
sense that emergency managers can concentrate only on those people who need 
assistance for evacuation.   
Evacuees from the areas outside the evacuation zone who leave despite the fact they are 
not threatened are also referred to as ?shadow evacuees?. This situation can pose 
significant problems, e.g. unnecessary traffic congestion, which will also delay or stop 
the actual evacuees from reaching safe ground. On the other hand, in the case of severe 
weather conditions, shadow evacuees can put themselves in greater danger than if they 
remain where they are.  
Depending on circumstances, the shelter or evacuation options can be used separately 
or in combination to provide the most effective response. Figure 1 shows the sequential 
phases of emergency management (MCDEM, 2008a). 
  
Figure 2.1 Phases of evacuation (MCDEM, 2008a) 
 
36 
 
 Decision to order an evacuation: The decision phase constitutes the period 
when data from the field is processed and a choice is made whether to order an 
evacuation or advise people to shelter-in-place.  
 Warning: During this phase, notifications are issued to the public, which contain 
an explanation of the situation and recommendations.  
 Evacuation: This phase describes the actual physical movement from an area 
 Shelter: This phase occurs when evacuees are accommodated in welfare camps. 
This phase usually includes the registration process, accommodating evacuees 
and assessing the provision of welfare and recovery requirements. 
 Return: This phase involves co-ordinating the physical return of evacuees to 
their homes following an assessment of the evacuated area and issuing an all 
clear.   
The duration of each phase differs according to the nature of the hazard, as do the 
emergency management activities and the requirements of decision making. For most 
emergency events, evacuation typically starts at the local level and then escalates as the 
impact of the hazard spreads. Similarly, whilst local authorities deal primarily with local 
community needs they must share situation information and resources and co-ordinate 
planning with national authorities. Mass evacuation cannot be executed by a single 
agency or organisation in isolation. Co-ordination between all the relevant stakeholders 
is important before and during all phases of evacuation. The number of organisations or 
stakeholders and their activities related to mass evacuation depends directly on the 
following factors (MCDEM, 2008a): 
 Geographic size of the region 
 Regional topography  
 Regional hazards and vulnerabilities 
 Demographics  
 Size and density of population 
 Resources available  
As the number of organisations increases, effective communication and co-ordination 
between them becomes very complex and challenging. The next section presents an 
37 
 
example of a tsunami scenario to illustrate the planning, information sharing, and 
decision making processes for successful mass evacuation.  
2.2.2 Mass evacuation in case of tsunami  
A tsunami is a series of waves that can move on shore rapidly, last for hours and flood 
coastal communities with little warning. Tsunamis can be triggered by a variety of 
geological processes such as earthquakes, landslides, volcanic eruptions or meteorite 
impacts (National Research Council, 2011). Since 1800, when modern record keeping 
started, tsunamis have taken many lives around the world. The threat of a potentially 
catastrophic tsunami on coastal areas looms in seismically active regions in the Pacific 
and Atlantic (Figure 2.2) (National Research Council, 2011). Recently, tsunamis 
generated by earthquakes in West Java (July 2006), Samoa (September 2009), Chile 
(February 2010) and Japan (March 2011) have flooded coastlines worldwide, 
highlighting the need for a focused and well-co-ordinated effort to minimise the loss of 
life and property. Tsunamis happen rarely enough to allow a sense of a false alarm, but 
when they do occur there may be just minutes or hours for people to reach a safer 
location National Research Council, 2011). 
 
Figure 2.2 Threat of a catastrophic tsunami on coastal areas of Pacific and 
Atlantic (source: USGS4) 
                                                 
 
4 http://vulcan.wr.usgs.gov/Imgs/Gif/PlateTectonics/Maps/map_plate_tectonics_world_bw.gif 
38 
 
A timely evacuation decision and the successful evacuation of people from an area at 
risk can save many human lives. Making a right and timely evacuation decision is based 
on the right interpretation of incoming situation information. Similarly, the decisions 
which task an emergency manager should carry out at the particular phase of an 
emergency depend on the goals and objectives of his/her role. Once received, the 
information is synthesised and interpreted to make sense of it. The decision is then 
made whether to evacuate or not. Once the requirement for evacuation is confirmed, a 
decision is made about the time to start evacuation. For this decision, seismic, Deep-
Ocean Assessment, and Reporting Tsunami (DART) data, along with the information 
on the characteristics of potential areas to be evacuated and available resources to 
complete the task are required.  
Moreover, an overlap in the understanding of a situation is required by emergency 
managers to achieve a consensus so that the evacuation decision can be made as quickly 
as possible. This overlap is achieved by sharing and discussing situational knowledge 
amongst the emergency team members. Once a decision is made, co-ordination among 
several organisations is needed to implement the decision. On a macro level, each of 
these organisations should know which organisation needs to co-ordinate with them 
about ?what they are doing? and ?what they need from them? and vice versa. On a 
micro level, individuals should know which of their team members are dependent (need 
information or require some actions to be performed) on them and vice versa. Without 
this knowledge, organisations and individuals will not be able to co-ordinate their 
activities to achieve the overall goal.  
2.2.3 Emergency management in New Zealand 
In New Zealand, the National Crisis Management Centre (NCMC) looks after a crisis at 
national level and co-ordinates with the regional emergency offices known as Group 
Emergency Operation Centres (GEOCs). GEOCs, in turn, co-ordinate with local 
Emergency Operation Centres (LEOCs) within that region. Co-ordination between 
NCMC and GEOCs and LEOCs is critical for decision making, as most of the major 
decisions are made by NCMC and implemented by LEOCs in co-ordination with 
supporting organisations such as the Fire Service, Police etc. New Zealand uses Co-
ordinated Incident Management System (CIMS) for managing the response to an 
incident involving multiple responding agencies. CIMS has five basic functions: 
Controller, Planning & Intelligence (P & I), Operations, Logistics and Welfare. In all 
39 
 
emergency operation centres, including NCMC, GEOCs or LEOCs, these functions 
exist either in the form of sub-teams or individuals, depending upon their scale of 
responsibilities as shown in Figure 2.3.  
 
 
 
 
 
 
Figure 2.3 CIMS Structure of LEOC, GEOC and NCMC (MCDEM, 2009) 
The P&I function is responsible for forecasting the incident development, anticipating 
likely needs, and drafting the Incident Action Plan. This role of the GEOC has a 
strategic scope. The Operations function enacts the Incident Action Plan, which makes 
sure that responders are as focused and aware as possible so they can fulfil the 
objectives set by the Controller. The Operations Manager is generally responsible for 
the operational command of resources. This means allocating specific functions to 
agencies in their areas of expertise, monitoring their performance and providing a 
communication link between the responders and the other elements of the GEOC, 
especially Logistics. The Logistics function or Manager ensures the continuity of 
operations by making sure there are sufficient resources on site and within GEOCs. The 
Welfare function is responsible for the wellbeing of affected people during or after the 
emergency situation, especially in the case of evacuees. They set up temporary welfare 
camps and provide accommodation, food and basic medical facilities until required. The 
Controller is responsible for the overall incident progression and has overall 
accountability for the incident. Moreover, the controller function approves the Situation 
Reports (SitReps) and Incident Action Plans (IAPs) issued by the GEO. 
Clear information sharing procedures and response prioritisations help in achieving a 
co-ordinated management of an emergency. Consistent emergency management 
requires information sharing among involved organizations along with specific event 
Controller 
Function 
Welfare 
Function 
Planning & 
Intelligence 
Operations 
Function 
Logistics 
Function 
40 
 
needs and priorities. Reliable information flows and proper prioritisation take place 
when organisations are well aware of the procedural platform to be used in emergency 
response and critical information required to make decisions is available. Hence, 
information requirements for the functions illustrated in Figure 2.3 for decision making 
and carrying out their tasks is vital for the successful evacuation.  
2.3 Emergency Decision Making  
2.3.1 An overview  
Decision making is the cognitive process of selecting one choice out of a set of choices 
(Grant, 2009). The process starts when a task is to be done, an action is to be taken, a 
goal is to be achieved or a need is to be fulfilled, and different options exist to carry it 
out. It ends when the decision maker has selected an option. The decision maker can be 
an individual, a team, or a coalition of organisations. In emergency response, an option 
is usually an intended course of action, with actions occurring serially or concurrently 
(Grant, 2009). Below are some of the important factors which affect emergency decision 
making. 
Emergency decision making is different: Decision making during emergency 
situations differs from ordinary or business decision making. Because lives and 
infrastructure may be at risk, stress levels are high, available time is short, the situation is 
continuously evolving and often information is incomplete and sometimes conflicting. 
In these circumstances, rapid situation assessment is a key pre-requisite for making 
appropriate decisions. Salas et al., (1995) describes this as a cognitive process in the 
situation assessment that provides the integration and understanding of information and 
temporal processes in which the experiences are used to predict the future. Stress being 
a conspicuous feature of emergency decision making, the degree of stress must also be 
linked to the process and outcome of emergency decisions (Bond & Cooper, 2005).  
The role of stress: Stress is defined as a psychological factor that threatens the 
individual?s physical and psychological well-being and increases the individual?s 
responsibility for the successful task performance (Driskell & Salas, 1991). Berkum 
(1964) attributed the decrease in working memory, where all the information is encoded 
for processing, as the cause of decreased performance under stress. Hence, excessive 
stress significantly reduces the human capability for information processing (Keina & 
Friedland, 1984). Fiedler and Garcia (1987) argued that decision performance is 
41 
 
influenced by the individual?s cognitive resources, level of intelligence, experience and 
technical expertise, which in turn are influenced by the degree of anxiety or stress. Task 
complexity and team support also have an influence on the decision outcome (Fiedler 
and Garcia, 1987).  
The role of experience: Decision making is based on a concept, an understanding and 
identification of the general requirements of the situation (Stevens & Campion, 1994). 
Klein (1989) defines the stages of decision making as feature matching, situation 
assessment, mental simulation of actions performed in similar past situations, and the 
outcome of action. Wickens (1992) argued that decision making is based on a decision 
maker?s experience and an increased ability to assess risk. Therefore, even if a situation 
is evolving, an experienced emergency manger is more able to recognize, interpret and 
integrate the new information and make the decisions about how the response should 
proceed. Klein (1989) found experienced people were making choices based on their 
prior experiences and their ability to recognise or classify a situation based upon the 
critical cues from the environment and their goals and expectations from typical actions. 
The most successful decision makers were the ones with a good understanding of their 
goals and recognised the prominent features of the situation. They recalled a single 
course of action based on the recognition of situations and evaluation of the options by 
mental simulations to assess the outcomes (Bond & Cooper, 2005). However, this 
model cannot be applied when a previous situation is not completely remembered.  
The role of available time: In emergencies, the tendency to make decisions faster 
makes them more prone to errors. Therefore, the stress level is also high when decisions 
are required in a short time period. Emergency situations are characterised by dynamic 
and evolving conditions, role uncertainty and complex situational constraints. Some of 
the several factors that contribute to this complex process are the unique attributes of 
disasters, rarity of disasters and less time during the disaster to understand and to obtain 
the experience necessary for emergency response. These problems are highlighted from 
the perspective of information management and decision making within integrated 
emergency management environments (Paton & Jackson, 2002). Emergency 
management needs to deal with these characteristics by delegation, communication, 
decision making and inter-agency co-ordination.  
42 
 
The role of co-ordination and collaboration: Inter-agency communication is 
important for effective decision making (Paton et al., 1999). In large scale emergencies, 
like the risk of a tsunami, a team of various individuals is involved in decision making. 
These individuals are usually from different backgrounds with different expertise and 
experience. Moreover, many of their objectives differ, depending upon their roles in the 
overall emergency management. Therefore, understanding the situation in a right and 
similar way is a challenge. Implicit information gathering is more effective if team 
members have a good understanding, not only of their own information needs, but also 
the needs of the fellow team members who depend on them for information sharing. 
Hence, for effective multi-organizational and multi-disciplinary co-ordination and 
performance, a team mental model is very important to provide the information 
required to achieve the common goal. With a team mental model, an implicit knowledge 
of activities and procedures called tacit knowledge exists within a team (Paton & 
Jackson, 2002). Tacit knowledge permits people to learn from their experience and to 
apply their knowledge in a goal-directed manner (Sternberg et al., 2000). Tacit 
knowledge also improves a person?s adaptability to the changing environment. 
Therefore, it is an important component of effective performance in evolving, uncertain 
and complex emergency situations.  
The role of explicit and tacit knowledge: The knowledge requirements for 
emergency decision making, as identified by Wang and Rong (2007), include explicit 
emergency knowledge sources and the agility of emergency knowledge support and 
features. The emergency knowledge sources include emergency environment 
information, contributing emergency incidents and emergency documentation. As the 
emergency knowledge required for decision making comes from diverse sources, it 
requires decomposition, matching and integration. The sources mentioned by Wang and 
Rong (2007) are quite important, but these authors ignored the other forms of 
knowledge, e.g. tacit knowledge (Paton & Jackson, 2002) which is experiential and 
obtained from exercises, trainings and simulations or from knowledge created about 
fellow team members? work and their shared mental models.  
Thompson and Dowding (2002) propose the following factors as the most critical for 
the decision making process. 
 The decision maker?s ability or cognitive resources  
43 
 
 The decision maker?s situational awareness (SA) 
 Level of stress  
 The difficulty of the task or its complexity  
 Level of team support  
 
These factors are strongly inter-dependent. For example, when a task is difficult or 
complex and the decision maker has a lack of situation awareness, the stress level rises 
(Sinha, 2005). Similarly, providing a higher level of team support can reduce high stress 
levels. The excessive stress also affects a decision maker?s ability to process information 
by reducing the short term memory (cognitive resource). Hence, situation awareness and 
team support can be improved to reduce the stress factor, ultimately improving the 
decision maker?s ability to make better and informed decisions. These issues are further 
discussed in subsequent sections.  
2.3.2 Models of emergency decision making 
In business, strategic leaders make critical business decisions after several hours or days 
of thinking though the options, and carefully evaluating each in turn against their 
business objectives using decision analysis methods. This is the classic method of 
decision making in which decision makers are trained to evaluate various options. 
However, in a dynamic and time critical situation, when a decision is needed quickly and 
many lives and infrastructure are at stake, the classical method of decision making fails. 
Usually there is not enough time to gather complete data and evaluate all possible 
options. Therefore, in emergency situations, people do not consciously evaluate options, 
but they know ?what is the right thing to do? (Flin, 1996). This can be called intuition, 
but when it comes to the judgements, highly complex mental activity happens in the 
mind of an individual. However, a question arises - can decision making in such critical 
situations be left for intuition and judgements alone. We will come back to this 
discussion later; let us first look at the two main categories of decision making (Kobus 
et al., 2001): 
 Intuitive or Naturalistic Decision Model 
 Analytical or Rational Decision Model 
It is argued by Klein (1997) that in an emergency situation decision makers select the 
best course of action based on their job role, experience and the situation in hand, 
44 
 
whereas in less stressful conditions decision makers tend to evaluate different 
alternatives (Orasanu & Connolly, 1993). Decision making techniques in an emergency 
context usually deal with cognition by using knowledge, experience, logical reasoning 
and mathematical analysis etc.  
State of the art works (Albert & Steinberg, 2011; Downs & Fischhoff, 2009; Luce & 
Winterfeldt, 1994; Reyna & Rivers, 2009) describe decision models as normative, 
prescriptive and descriptive. Normative models describe how decisions should be made. 
Prescriptive models describe how decisions could be made and take a rationalist view 
based on decision analysis. In contrast, descriptive models describe how decisions are 
made from either an interpretive or a naturalistic perspective.  Interpretive process 
includes hypothetical-deductive processes (Thompson & Dowding, 2002) which vary 
depending on the situation and expertise cognitive continuum (Hamm, 1988).  
In an emergency management context, these decision models are based on the classic 
frameworks such as Observe-Orient-Decide-Act (OODA) cycle (Boyd, 1987), and 
Decision and Learning Cycles (DLC). Both of these theoretical frameworks provide the 
necessary flexibility to understand emergency decision making (Simon, 1955).   
Normative decision model aligns with classical decision theory. According to this 
model, the decision maker is believed to be a fully informed rational person who 
examines a set of options, or course of actions, and weighs various attributes in order to 
make the best choice. The more recent naturalistic decision model describes complex 
cognitive functions, which can work in highly demanding situations such as those 
having: 
 Short time 
 Uncertainty 
 Unclear goals 
 Team and organisational expectations 
 Dynamic situations 
 High stakes like lives and infrastructure.  
The usual methods provided by the normative decision making i.e. deductive logical 
analysis, evaluation of probabilities and statistical methods do not seem as useful during 
highly stressed emergency situations due to a limited short term memory (Kaemps et al., 
45 
 
1993; Klein, 1998). Normative decision models are more useful for well-defined 
problems in which there is plenty of time to evaluate options under limited stress 
(Endsley & Jones, 1997). The following sections summarise the fundamentals of these 
decision models and their suitability for emergency decision making.  
2.3.3 Intuitive or Naturalistic Decision Model 
Classical decision making is primarily built around analytical and heuristic approaches to 
decision making (Hutton & Klein, 1999). It is argued that being largely normative 
instead of descriptive leads to failure in capturing critical aspects of how experts actually 
make decisions in emergency situations (Endsley et al., 2007). Naturalistic decision 
models (NDM) remain the most cited and popular in most of the cognitively 
demanding situations (Sinha, 2005; Endsley et al., 2007). The distinctive feature of 
naturalistic decision making is that the decision makers are highly proficient and 
experienced in their domain (Elstein, 2001). Expert decision makers use the situational 
assessment, which makes use of signal encoding, goal assessment, expectancy checks 
and appropriate actions; a rapid encoding of stimuli based on experience and decisions 
are rule based rather than knowledge based (Flin et al, 1996).  
NDM research has been done for a variety of domains such as nuclear power plants 
(Roth, 1997), commercial and military pilots (Amalberti & Beblon, 1992), fire 
emergencies (Klein, 1999; Klein & Klinger, 1991) air traffic control (Rodgers et al., 
2000). Although these domains are contextually different, they are also similar in many 
ways e.g. uncertainty, complexity, significant consequences, lives at risk, competing goals 
and time constraints (Zsambok & Klein, 1997). The Recognition Primed Decision 
(RPD) model was introduced to explain the NDM process during complex situations 
(Klein et al., 1986; Klein, 1989; 1993). The use of RPD is evident from many emergency 
situations in which the response also requires the appropriate assessment of a situation 
for decision making. The RPD model focuses on situational assessment rather than 
option assessment, stressing the significance of experience in decision making and 
relying on ?satisfactory? or ?good enough? decisions (Simon, 1989). Such decisions are 
likely when time pressure is high. However, it is clear that good decisions are dependent 
on experience and expertise. Situation recognition allows the decision maker to classify 
the tasks as well known. The familiarity carries with it the recognition of the following 
types of information:  
46 
 
 High priority goals (Examples: evacuating the areas which are under high and instant risk 
first),  
 Cues to monitor (Example: speed, direction and height of tsunami waves at different time 
instances and locations),  
 Expectations about the unfolding situation (Example: expected height of wave at 
certain locations, expected arrival time of first wave on certain locations).  
According to Klein and Klinger (1991), the RPD model can be summarised as follows: 
 Options are generated consecutively, and the more usual course of action is 
considered first. 
 Option evaluation is also performed consecutively to analyse the adequacy of 
the option and to identify possible problems and solutions for them. 
 The RPD model includes facets of problem solving and evaluation along with 
decision making. 
 Experienced decision makers are able to respond quickly by identifying a tested 
course of action at the start, rather than generating and evaluating a large set of 
options.  
The RPD model works in three possible ways (Klein & Klinger, 1991). The simplest 
way of RPD working is when the situation is recognised successfully and the respective 
obvious reaction is implemented. A more complex way is applicable when the decision 
maker consciously evaluates the reaction, using imagination to uncover problems prior 
to carrying it out, and minor changes are made to the course of action. In the most 
complex case, the evaluation reveals problems, which require modification, or the 
option is judged to be flawed and is rejected in favour of the next most usual useful 
reaction.  
2.3.4 Rational or Analytical Decision Model 
A rational decision model (RDM) is used when the best immediate option is selected 
(Simon, 1955) by a decision maker who is called the rational decision maker. The word 
?best? shows that the decision is, or should be, most favourable. According to Grant 
(2009), to find the best option, each one of the complete set of possible options must be 
given a score, then the set must be ranked based on this score, and the highest-ranking 
option is selected. The score given to an option is composed of the values of its 
47 
 
attributes. Hence, RDM requires that the decision maker must have perfect information 
about the values of all attributes of all the possible options. The word ?immediate? 
means that the decision maker considers the current problem is influenced neither by 
any related decisions made in the past nor by any possible future consequences of this 
decision (Grant, 2009). Hence, this model rules out using any knowledge gained from 
previous experience to make decisions and RDM research focuses on the option 
selection. This is based on some judgment or logical plan such as the maximisation of 
the expected value or the removal of options by features or aspects (Dastani et al, 2003).  
The selection of best decision makes RDM attractive at first. However, it also has some 
limitations. Knowledge about RDM has been largely gained by performing experiments 
in laboratory settings using immature subjects (Klein, 1998). As we know, the rational 
decision maker must have a perfect knowledge and when the number of options and 
their related attributes is large, the process may take a substantial amount of time and 
resources. The description of RDM shows that any system, which is designed for 
supporting rational decision making, should have the following properties (Grant, 2009):   
 A set of known options with their attributes;  
 A mechanism for obtaining the values of the attributes;  
 An algorithm for computing each option?s score from its attribute-values;  
 An objective method for selecting the highest ranking score.   
The lengthy and tiring nature of RDM means that it is applicable to domains in which 
the goals are well defined, the environment changes slowly, no other active agents are 
present and enough time is available to evaluate all available options, which makes it 
very inadequate to be used during emergency situations that require the opposite. 
The limitations of pure NDM and RDM can be minimised by combining the two 
approaches. Bryant (2002) also proposed a synthesis of NDM and heuristic approaches 
to overcome the limitations of pure NDM.  However, to date very little research has 
been done combining both classical and naturalistic paradigms to explain decision 
making in complex situations. This literature gap is being addressed in the current study; 
we are considering computer support for combining these paradigms - more details of 
this combined approach will be provided in upcoming chapters.  
48 
 
2.4 Situation Awareness (SA) 
The first step in any decision model is understanding what is going on.   This 
understanding is required by the decision maker to suggest tasks that can achieve a 
decision?s objectives. A decision maker?s assessment of a situation results in a product 
called Situation Awareness (SA) (Endsley, 1995b). The SA has emerged as a key 
cognitive construct of interest, formed from the terminology and challenges of the 
aviation field (Endsley et al., 2007). For an operator, SA is a mental representation of 
the world around him/her at any given point of time. In simple words, SA is about 
knowing and recognising what is going on around us (McIlvaine, 2007). It is required 
for moment-to-moment decision making and improved performance in complex 
systems (Endsley, 1988a). Nowadays, SA research has been extended to many other 
sectors like air traffic control, nuclear and petro chemical plant operation, military 
contexts, driving aviation, emergency response (Endsley, 2000; Stanton et al., 2005).  
2.4.1 SA definition 
SA has several definitions. For example, according to Smith and Hancock (1995) it is an 
externally directed consciousness that directs actions in a dynamic tasks environment.  
Whereas, according to Dominguez (1994) SA is also integration of information with 
previous knowledge to form a coherent mental picture and anticipating future events. In 
contrast, Taylor (1990) believed, SA is the knowledge, cognition, and anticipation of 
events, factors and variables affecting the safe, expedient and effective conduct of the 
mission. Sarter and Woods (1991) defined SA, based on the integration of knowledge 
resulting from recurrent situation assessments, but did not explain situation assessment. 
Other researchers have proposed alternative definitions of SA (Adams et al., 1995; 
Billings, 1995; Durso and Gronlund,1999; Stanton et al., 2005; Salmon, et al., 2008). The 
range of definitions arises in part from the complexity of different cognitive models and 
in part from alternative views that emphasise particular features of a situation. Also, 
some definitions are broad, whilst others try to present a structured interpretation of the 
assessment process.  
Therefore, SA has a range of theoretical connections emanating from the work of 
Adams et al. (1995), Smith and Hancock (1995), and Endsley (1995). A universally 
accepted definition for SA has consequently yet to emerge (Stanton et al., 2005) but two 
theories of SA ? the three level theory of Endsley (1995) and the perceptual cycle model 
proposed by Smith and Hancock (1995), have emerged as dominant (Stanton et al., 
49 
 
2005). Endsley?s (1995) definition of SA has been applied in many domains (Wickens, 
2008) and has become the most frequently used and extensively cited theory in the field 
(Albers, 1998; Klein, 2000; Son et al., 2007; Stanton et al., 2005). Moreover, Endsley has 
presented the best information processing approach compared with other models 
(Stanton et al., 2001).  
2.4.2 Endsley?s three level model of SA 
SA has been entailed as playing a conspicuous role in several emergency decision 
making strategies (Endsley, 1995a; Endsley, 1995b).  According to Endsley (1995b), SA 
is a key facet of effective emergency decision making and operational performance. SA 
is mostly defined in operational terms. For a given operator, SA is defined in terms of 
goals and decision tasks for some specific job (Endsley, 2000). A formal and widely 
accepted definition of SA that has been found applicable across a wide variety of 
domains is therefore: 
 ?The perception of the elements in the environment within a volume of time and space, the 
comprehension and the projection of their status in the near future? (Endsley, 1988a).  
Perception is the first level of SA, it means understanding the importance of information 
about any situation. For example, in the context of a tsunami; earthquake magnitude, 
source location, depth and wave height and direction, if the earthquake source is in the 
water are critical attributes.  
Comprehension is the second level of SA, it constitutes how people combine, interpret, 
store and retain information.  Comprehension also includes integrating multiple pieces 
of information from SA Level 1 and determining their meaning and relevance to the 
operator?s goals. For example, combining the earthquake mangnitiude with the 
geograhical co-ordinates of an earthquake source, determines whether the earthquake is 
powerful enough and offshore to cause a tsunami or not. Moreover, processing the 
wave height information, if the source is offshore, it can be used to determine whether a 
tsunami has been generated or not.   
Projection is the third and highest level of SA; it is the ability to forecast future situations 
from existing and previous situations. This ability allows timely decision making to 
counter any future risk. For a tsunami, forecasts would include predicted wave heights 
on arrival at the coasts, arrival time of waves, indication of areas under threat etc.  
50 
 
SA is critical to optimal decision making, but there are many other factors which play an 
important role. Klein?s work in the area of Recognition Primed Decision making 
provides strong evidence for the impact of situation recognition/classification and 
associated action selection (Klein, 1989; Klein, Calderwood & Clinton-Cirocco, 1986). 
The relation between SA and decision making is also discussed by Adams et al. (1995) 
and Smith and Hancock, (1994). SA informs decisions and SA is formed by feedback 
from decisions. Therefore, SA, decision-making and performance can be theoretically 
perceived as distinct stages that can affect each other and can be decoupled through 
other factors (Endsley, 2000).  
Endsley (1995b), as shown in Figure 2.4 below, explained sources of SA information. 
Decision makers can directly observe the situation and they can use technology or 
systems e.g. internet, telephone, radios etc. or group members to collect subsets of 
available information. Decision makers perceive and interpret information that results in 
SA.  
 
Figure 2.4 Information sources of SA (from Endsley, 1995b) 
Although SA has a very significant impact on decision making, good SA does not 
guarantee optimal decisions (Son et al., 2007).  Many other factors such as a change in 
the situation before the implementation of a decision, strategy, experience, training and 
personality, organizational and technical constraints may also affect the decision making 
process (Endsley & Garland, 2000). However, during emergencies, even experts can 
make incorrect decisions due to insufficient SA at different levels (Son et al., 2007).  
51 
 
2.5 Situation Awareness in Collaborative Systems 
In larger events, several organisations are involved in emergency management processes 
so that the co-ordination of decision making is a challenge. Team members have 
different characteristics such as cultural background, environment, educational 
background, experiences and goals, depending on their roles, and it can be difficult to 
draw together individual expertise to handle a problem. Therefore, understanding the 
situation correctly and in a shared, common way is difficult. In addition, in an 
emergency, information is frequently evolving, uncertain and incomplete so that 
information co-ordination5 requires information to be current and available in a directly 
usable form for accurate team decisions.  
Due to the major requirement of team work in current disaster management 
organisations, the concepts of shared and team situation awareness (SSA & TSA) are 
currently receiving enormous attention from both human factors community and 
disaster management (Fiore et al., 2003).  
2.5.1 Shared Situation Awareness (SSA) 
Shared Situation Awareness (SSA) refers to the extent of common SA requirements of 
team members (Endsley & Jones, 2001). Each team member has specific SA 
requirements for their task, some of which are common with their team member?s 
requirements, and these researchers define SA as, ?the degree to which team members 
have the same SA or shared SA requirements? (Endsley & Jones, 2001).  
 
Moreover, Endsley and Robertson (2000) suggest that successful team performance 
requires that individual team members have a good SA of their own elements and the 
same SA for those elements that are shared. Thus, SSA refers to the overlap between 
the SA requirements of the individual team members. This concept is illustrated in the 
Venn diagram of Figure 3 where the circles represent each team member?s SA 
requirements, and the overlapping segments depict the SSA (Endsley & Jones, 2001). 
The white areas of the circles indicate the information that does not need to be shared. 
                                                 
 
5 Co-ordination of people and organisations are also priorities in emergencies. 
52 
 
Sharing information that is not relevant to the SA requirements of other team members? 
roles would only create "noise" and wasteful effort to acquire it. 
 
 
 
 
 
 
Figure 2.5 Shared Situation Awareness (Endsley & Jones, 2001) 
2.5.2 Team Situation Awareness (TSA) 
A team is defined as ?two or more people dealing with multiple information resources, 
who work to accomplish some shared goal? (Salas et al., 1995). Team Situation 
Awareness (TSA) is more complex than individual SA. Salas et al. argued that there is a 
lot more to TSA than just combining team members? SA. Moreover, due to the many 
other cognitive, social and team factors, research into the construct is challenging, scarce 
and difficult and there is no consensus on exactly how TSA works.  
Taking an operational view, the most widely used model of SA due to Endsley and 
Jones (2001) leads to a definition of TSA as ?the degree to which every team member 
possesses the situation awareness required for his or her responsibilities? (Endsley, 
1995a). Endsley argues that the success or failure of a team depends on the success or 
failure of each of its team members. At its simplest level, therefore, each team member 
has a sub goal related to his/her role that feeds into the overall team goal. A set of SA 
information elements is linked with each member's sub goal and these are shared with 
other members of the team. The shared subset of information is the basis of team co-
ordination which may occur directly or be mediated by technology (Endsley & Jones, 
2001). TSA can therefore be represented as a Venn diagram (Figure 2.6) in which the 
circles represent team members? information relevant to their roles and the overlapping 
areas represent their shared information. A two-circle overlap denotes shared situation 
awareness (SSA) and the three-circle overlap signifies team situation awareness (TSA). 
53 
 
Teams with more than three members would clearly require diagrams with additional 
circles and multiple overlaps.  
 
 
 
 
 
Figure 2.6 TSA: the goals and SA requirements of all team members (Endsley & 
Jones, 2001). 
Endsley & Jones (2001) have extended their operational view in a model that describes 
the factors that help to build SSA and hence TSA (Figure 2.7). These factors ? 
requirements, devices, mechanisms, and processes ? facilitate interaction between the 
participants and the sharing of information so that teams can develop high levels of SSA 
and TSA. 
 
 
 
 
 
 
 
 
 
Figure 2.7 Factors that build high levels of SSA and TSA (Endsley & Jones, 2001) 
Taking a different perspective, Salas et al. (1995) have argued that there is a lot more to 
TSA than just combining team members? SA. Apparently, TSA is multi-faceted, 
Team SA 
Requirements 
Information needs 
Team task status 
Team SA Devices 
Direct communication 
Shared displays 
 
Team SA Processes 
Sharing SA information 
Check conflicting 
information 
f k
Team SA Mechanisms 
Shared mental models 
Projections of team 
54 
 
comprising an individual team member?s SA, shared SA between team members and the 
combined SA requirements of the whole team. In this framework, TSA is mostly 
understood as a shared understanding of the same situation (Nofi, 2000) whereas, 
according to Perla et al. (2000), when we all understand a given situation in the same 
way it is SSA.  
Salas et al. (1995) suggest that TSA comprises two important, but badly understood 
processes, individual SA and team processes, and depends on communication at various 
levels of SA. For example, perception of elements at level one SA is effected by 
communication of mission objectives, individual tasks and roles, team capability and 
team performance factors. Similarly, comprehension at SA level two is impacted by the 
interpretations made by other team members, so it is evident that sharing an individual?s 
SA leads to the development or modification of another?s SA. In addition, the mental 
schema limitations of an individual can be offset by information exchange and 
communications (Salas et al., 1995).  
Salas et al. (1995) consequently defined TSA as ?the shared understanding of a situation 
among team members at one point in time?, and concluded that ?team SA occurs as a 
consequence of the interaction of an individual?s pre-existing relevant knowledge and 
expectations, the information available from the environment and cognitive processing 
skills that include attention allocation, perception, data extraction, comprehension and 
projection?.  
Another approach due to Wellens (1993) uses a model of distributed decision making to 
describe SA during collaborative activity and suggests that the key to TSA lies in the 
arrangement of teams to provide adequate separation for individual SA acquisition with 
sufficient overlap for co-ordination. Wellens (1989) defines TSA as the ?sharing of a 
viewpoint between two or more individuals regarding current events in the 
environment, their implications and projected future?.  
The TSA definition used in defining this framework is provided by Shu and Fruta 
(2005), which is:  
?Two or more individuals share the common environment, up to the moment understanding of situation 
of the environment, and other person?s interaction with the cooperative task.? 
55 
 
According to Shu and Furuta (2005), TSA includes two basic elements, an individual?s 
SA and mutual awareness. Mutual awareness refers to the awareness that individuals of a 
co-operative entity have of each other?s activities, beliefs and intentions. For example, if 
two team members A and B want to understand and act on a situation co-operatively, 
each of them needs to recognise the parts of the situation that he/she is responsible for, 
and to believe that the other member recognises their own parts of the situation and is 
willing to co-operate. In addition, each member must contribute a belief about the other 
team member?s understanding. Hence, this definition covers both the individual and 
shared understanding of the situation and the execution of co-operative tasks. In short, 
for the team of two members A and B, A?s TSA consists of three basic components (1) 
A?s own SA, (2) A?s belief about B?s SA and (3) A?s belief of B?s belief about his/her 
own (A?s) SA. If any of these is missing, the TSA is incomplete. 
The complexity of interaction clearly increases with increasing team size and much work 
needs to be done to refine definitions and terminology, unravel interactions, and 
synthesise a complete picture of TSA. Only then, for example, will it be possible to 
answer questions such as ?how much sharing of information (extent of overlap of 
circles, e.g. in Figures 2.5 and 2.6) is needed to ensure successful completion of a 
collaborative task in a given situation?? 
2.6 Information System Support for Emergency Management   
Many researchers (e.g. Carver & Turoff, 2007; Jennex, 2007; Manoj and Baker, 2007; 
Turoff et al., 2004; Turoff, 2002) have concluded that in emergency situations, decision 
making is improved by dynamic information retrieval, and sharing and presenting 
information in the right format, to the right person, at the right time. The faster the 
decision makers get the required information, the more effective their response will be 
(Van de Walle & Turoff, 2007). During emergency decision making, lives and 
infrastructure are at stake and the level of stress is very high so that the importance of 
an emergency response system to assist the decision making process becomes obvious 
(Bellardo et al., 1984; Jennex, 2007).  
However, very few systems seem to contribute noticeably to the emergency decision 
making process (Van de Walle & Turoff, 2007). One fundamental reason is that systems 
are sometimes designed without considering the users (de-Leoni et al. (2007). Even when 
users are involved in the design and development there is often a lack of understanding of 
56 
 
human decision making requirements (Carver & Turoff, 2007). Carver and Turoff actually 
argue that an information system should essentially work as a team member to provide 
enough decision making support during an emergency.  
2.6.1 Emergency information systems frameworks 
Bellardo et al. (1984) argue that an emergency response system should consist of several 
basic components; a database, data analysis capability, normative models and interfaces.  
However, this model fails to address issues such as consideration for emergency plan 
and emergency response infrastructure, co-ordination between multiple organisations, 
information integration from diverse sources, knowledge from past emergencies and 
integrating multiple systems (Jennex, (2004). Jennex (2004) extended the emergency 
response system model of Bellardo et al., (1984) to facilitate the effective 
communication and data management to avoid information overload and argues that 
under stress emergency decision makers need a system that does more than just provide 
data (Jennex, 2007).  
Turoff et al. (2004) also acknowledged the necessity to improve the design and 
functionality of emergency decision support systems (EDSS) and developed a set of 
general supporting design principles and specifications for a ?Dynamic Emergency 
Response Management Information System? (DERMIS). DERMIS addresses the 
communication and information needs of command and control staff. Based on the 
emergency experiences emergency managers? needs, Turoff et al. (2004) also emphasise 
the system support for SA and suggest the use of templates that can be modified as 
needed. Lee and Bui (2000) also supported a template based emergency response system 
model where each template provides a standard set of placeholders or slots for 
representing the details of the activity. As these frameworks are only capable of 
providing decision support based on historical data or previous experiences, they lack 
providing support to process dynamic information or information processing in real 
time or in new situations.   
Because of the in-depth need analysis of stakeholders related to the use of technology, 
Meissner et al. (2002) proposed detailed communication architecture to support a basic 
emergency situation. In an extension to their work, Meissner et al. (2006) introduced 
conceptual information systems architecture MIKoBOS, suitable for supporting three 
different levels of organisational hierarchy. Therefore, Meissner et al., (2002; 2006) also 
57 
 
highlight the importance of understanding the specific needs of emergency managers 
while proposing a system-based support for emergency management. 
Courtney (2001) suggested a new decision style, as presented by the Singerian approach 
(Churchman, 1971), which incorporates other inquirers into the environment. The goal 
of the Singerian inquirer is to create verified common knowledge. Courtney (2001) 
recommended the use of this type of inquiring and unbounded system thinking (Mitroff 
& Linstone, 1993), to address the decision making in complex contexts. In the Singerian 
perspective, problem and knowledge domains are closely connected by bringing in the 
multiple perspectives or world views. The unbounded system philosophy apprehends 
the organisational (social) and individual (personal) viewpoints. All this leads to a novel 
decision making model which also discusses many other important aspects other than 
technical ones.  
Hwang et al., (2007) introduced a mobile-based ad hoc information system architecture 
suitable for Community Service emergencies. de-Leoni et al. (2007) proposed a 
communication architecture known as ?WORKPAD? to support decision making 
during emergency situations. ?WORKPAD? is a design based on task requirements 
identified via a task analysis technique called Hierarchical Task Analysis in various 
emergency scenarios. However, ?WORKPAD? does not support the individual needs 
and is limited to the common needs of team leaders. Kwan and Lee (2005) base their 
emergency response information systems architecture on a Global Information System 
(GIS), which makes use of information collected from intelligent transportation and 
building management systems. Neither of these technology-oriented studies has 
identified the contextualised end user requirements.  
2.6.2 Emergency management information systems (EMIS) 
Much of the research on emergency decision support systems has focused on different 
technologies. Examples include - transportation security decision-making for emergency 
response (Yoon et al., 2008), intelligent mobile decision support for triage 
(Padmanabhan et al., 2006), advanced knowledge models to support environmental 
emergency management (Hernandez & Serrano, 2001), aggregating knowledge of 
geographically separate experts (Mendonca et al., 2000), flood emergency decision 
support system (Mirfenderesk, 2009), emergency decision support system (Cortes et al., 
58 
 
2000), agent-based environmental emergency knowledge system (Liu, 2004), and expert 
system for chemical incidents (Yeh & Lo, 2004).  
Other research has concentrated on improving communication technologies. Examples 
embrace work by Otim (2006), Rosen et al. (2002), Potter et al. (2007), Iakovou and 
Douligeris, (2001) and McGrath et al. (2003).  
An ambitious venture, the ALADDIN project, aims to model, design and build 
decentralised systems that can bring together information from a variety of 
heterogeneous sources to take informed actions (Adams et al., 2008). The ALADDIN 
design considers some different aspects such as data fusion, decision-making, and 
machine learning etc. (Adams et al., 2008). Jang et al. (2009) and Lien et al. (2009) have 
proposed RISED, a ?Rescue Information System for Earthquake Disasters? for 
response following an earthquake.  
All these projects promote technology to support emergency decision making during 
various natural disasters. However, none of them can be said to adopt a particularly user 
centred approach to system design to understand end user requirements and to involve 
users centrally in system design and development. 
In contrast, Gao et al. (2007) proposed a user-centric system in a project called AID-N 
(Massey et al., 2006) that seeks to identify the unmet needs of triage response teams for 
medical assessment in an emergency. Aziz et al. (2009) also adopted a user centred 
design to develop a prototype for use in fire rescue operations. More recently, Katuk et 
al. (2009) used a knowledge acquisition approach to involve responders in the design 
and development of a web-based prototype for response in the aftermath of a flood.  
2.6.3 EMIS support for SA 
During emergency situations, as indicated, the information sources for each emergency 
manager that form their SA (and hence SSA and TSA) can be different depending on 
their role, goals, and tasks in hand (Son et al., 2007). The challenge is to integrate 
information rapidly from diverse sources, e.g. from direct observations of environment, 
team members, past experiences, emergency management plans etc. (Endsley, 2001) and 
present it in a useful way. In dynamic and time critical situations, it is usually not feasible 
to acquire SA only by direct observations from the environment (Son et al., 2007). In 
such circumstances, information systems can play a very important role in supporting 
59 
 
data collection, integration, processing, sharing, communication, and presentation 
according to the task requirement (Son et al., 2007). Endsley et al. (2003b) suggest that if 
an information system is designed to enhance the SA of its users, then such a system 
should take a user centred design approach. Some existing systems, which are based on 
SA requirements, are as follows. 
Lanfranchi and Ireson, (2009) proposed a project named ?WeKnowIt? that allows both 
emergency responders and community citizens to participate during flooding incidents. 
WeKnowIT is one of the few studies that recognises the importance of enhancing SA 
for better decision making during emergency response. However, the authors do not 
clearly explain their understanding of SA and hence the SA requirements captured were 
very generic.  
?WIPER? is a prototype emergency response system capable of generating traffic 
forecasts and emergency alerts for engineering, public safety and emergency response 
personnel (Madey, et al., 2006). WIPER is an attempt to explore the possibility of 
improving SA using mobile phones. However, neither end user requirements are 
reported nor the method of capturing end user requirements is clearly mentioned.   
Son et al., (2007) presented a comprehensive framework to ensure the effectiveness of 
disaster response operations. The framework aims at improving the SA while 
supporting collaboration among various civil engineers? roles. The study focuses on 
interaction between multiple individuals with an ultimate goal to improve SA in 
complex environments. Though the study (Son et al., 2007) is focused on the role of 
civil engineers, the work explains how a high level of SA can improve performance in 
dynamic and chaotic situations.  
Blandford and Wong (2004) recommend the SA-oriented system design for senior 
emergency medical dispatch operators. During their study, it was identified that 
ambulance operators work under extreme time pressure with high risks and have to 
make decisions with incomplete information. They adopted two strategies to understand 
user SA requirements; contextual inquiry interviews (Beyer & Holtzblatt, 1998) and 
critical decision method (CDM) (Hoffman et al., 1998) which are cognitive task analysis 
methods. As a result, a set of design guidelines for system displays to support 
emergency medical dispatchers was identified.  
60 
 
Prasanna et al. (2009) identified the information required for SA development during 
rescue and response operations in urban fire scenarios. Two techniques inherited from 
cognitive task analysis (CTA); Goal Directed Task Analysis (GDTA) and Applied 
Cognitive Tasks Analysis (ACTA) were combined to create a tool called Goal Directed 
Information Analysis (GDIA). Prasanna et al. (2009) successfully identified information 
requirements for all three levels of SA. Mock-ups were developed and validated but no 
performance test for the users was carried out to evaluate the usefulness of the design. 
However, the information requirements gathered based on SA were impressively 
detailed.  
Ferreira et al. (2009) used a task analysis technique in the form of a game based tool to 
collect the end users? needs for developing SA during emergency situations on a 
highway. The technique used by Ferreira et al. (2009) is promising for acquiring very 
detailed user requirements although few were reported. In addition, usefulness of the 
technique and validity of identified requirements were not put to the test.   
Thus, all these studies stress the importance of considering the end user needs while 
designing a system. These studies also highlight the complexity of extracting the user 
requirements operating in time critical and complex situations, and then emphasise the 
significance of choosing appropriate techniques to complete requirements elicitation. In 
an iterative process of design and development, most of these studies produced early 
prototypes. Some of the prototypes were also used to demonstrate feedback on the 
interfaces and display but none of them has been tested for the users? performance and 
the validation of identified requirements. The studies based on an SA-oriented design 
approach support the user centred design of systems and emphasise the importance of 
improving SA as an essential requirement for better decision-making in complex 
environments like emergency situations. However, it is often not clear from the 
published studies whether requirement elicitation applies to whole situations, specific 
tasks, roles, or contexts.  
There is very little in the literature that endeavours to understand the comprehensive SA 
requirements of emergency managers who work in such complex domains and engage 
in the high level collaboration that is required within and between various teams. 
Similarly, little has been done on the design of systems while considering the 
requirements of the shared and team situation awareness. Furthermore, there is not a 
61 
 
single study that has recognised the impact of SA on decision making for large scale 
emergencies such as tsunami risk for a highly populated area or city. Given the rise in 
the reported incidences of massive tsunamis and earthquakes, it is also surprising there 
is very little empirical work on the design and development of SA oriented information 
systems specifically aimed at improving the SA of emergency decision makers for mass 
evacuations in these catastrophes.  
2.7 Ontology 
2.7.1 Definition 
A parallel trend in emergency systems design within the past decade has been the use of 
ontological descriptions to model disaster scenarios and responses and attendant system 
design (Hoogendoorn et al., 2005; Little & Rogova, 2005; Liu & Fang, 2006; Wenjun et 
al., 2005). In philosophy, ontology is defined as: ?an explicit specification of conceptualisation? 
(Gruber, 1993). Alternatively, Studer et al. (1998) define ontology as ?a formal, explicit 
specification of a shared conceptualisation?. Conceptualisation refers to an abstract model of 
some phenomenon in the world by identifying the relevant concepts of that 
phenomenon. Explicit means the type of concepts used, and the constraints on their 
user are explicitly defined.  The term ?Formal? refers to the fact that the ontology should 
be in a standard language, which is machine-readable. ?Shared? reflects the notion that 
ontology captures a consensus, that is, not the private idea of some individual but the 
accepted and shared understanding of a group. In practical terms, this means that an 
ontology is a structured set of concepts describing a specific domain of knowledge that 
can be used to create a knowledge base for that domain. As a ?conceptualisation of 
shared knowledge?, an ontology can provide a common vocabulary that is equally 
understandable by all participants ? a valuable asset when domain users follow different 
terminologies.  
 This means that ontology is a hierarchically structured set of concepts describing a 
specific domain of knowledge that can be used to create a knowledge base. Ontologies 
can be used for many different purposes and applications and they can be constructed 
and organized in many different ways (Albertsten & Blomqvist, 2007).  
The information systems literature contains many definitions of ontology, some of 
which appear contradictory due to the use of different terms and loose application. One 
coherent and well-accepted definition (Noy & McGuiness, 2001) holds that an ontology 
62 
 
contains concepts (classes in ontological terminology), which have distinguishing 
features or attributes (slots) that impose constraints (facets) on their use. The relations 
between the concepts structure them into hierarchies according to various rules and 
axioms (Guarino, 1998) so that child concepts inherit (subsume) the attributes of their 
parents. An ontology, together with a set of individual instances of classes, constitutes a 
knowledge base. Though we will discuss the specific reasons to use ontologies in our 
system in later chapters, some of the general reasons to use ontology in information 
systems are as follows (Noy & MacGuinness, 2001):  
 To share common understanding of the structure of information among people 
or software agents 
 To enable re-use of domain knowledge 
 To make domain assumptions explicit 
 To separate domain knowledge from the operational knowledge 
 To analyse domain knowledge 
2.7.2 Ontology-based situation awareness (SA) 
Ontology is commonly understood on two levels: (a) as a knowledge base for a specific 
domain and (b) as a vocabulary, which uses appropriate terms to describe domain 
entities or agents, and the protocols for the relationships and interactions between them. 
Ontology therefore describes the domain knowledge and provides a consistent 
understanding that can be shared by users. If the domain is an emergency situation, then 
an ontological description can conceivably facilitate shared situation awareness.  
Several studies (Mathues et al., 2003; McGuinness et al., 2003; Horney et al., 2003; 
Boury-Brisset, 2003) have proposed ontology based SA systems. They also provide 
higher level situation ontologies to capture generic situation descriptions that require 
more domain specific knowledge for implementation.  
Ontologies provide considerable value in terms of data and knowledge acquisition, 
semantic integration (Baumgartner et al., 2008), semantic inter-operability and decision 
support. Moreover, ontologies provide the basis for a common semantic frame of 
reference for user groups from different backgrounds with different sub-goals that must 
operate in a collaborative way to achieve an overall larger goal. Ontologies are therefore, 
63 
 
considered a vital element of systems that aim to provide shared understandings of a 
domain and support shared situation awareness (Smart et al., 2005).  
2.7.3 Existing ontology-based systems to support situation awareness (SA) 
Several major projects like ?BeAware? (Baumgartner et al. 2010), ?AKTiveSA? (Smart 
et al., 2005), ?Information fusion for Natural and Man Made Disasters? (Rogova et al., 
2006) have successfully used ontology-based SA in real-life situations. The importance 
of ontology-based information fusion for enhanced situation awareness has been 
explained by many researchers  (Baclawski et al., 2003; Matheus et al., 2003; Matheus et 
al., 2004; Matheus et al., 2005; Sycara et al., 2003). Among these, Matheus et al. (2005) 
used ontology for formally defining situations. Based on their definition, they developed 
generic concepts that can serve as the basis for development of domain specific 
ontologies. They named the proposed system as situation awareness assistant (SAWA) 
as it is intended for enhanced situation awareness. SAWA also uses a number of 
semantic web technologies, namely Web Ontology Language (OWL), Semantic Web 
Rule Language (SWRL) (Fikes et al., 2003). The idea of SAWA is to provide a domain 
ontology that captures the objects and relationships of interest in a domain. The user 
controls the system situation monitoring functionality while the computation of 
relationships is delegated to the system and automated knowledge processes that reduce 
operator workload.  
BeAware (Baumgartner et al. 2010) is an ontology-driven framework aimed at increasing 
a user?s SA in traffic emergencies. A domain-independent SA ontology (Matheus et al., 
2005) is extended by incorporating spatio-temporal primitive relations between 
observed real-world objects. Moreover, ontology is made an integral part of BeAware 
throughout its architecture, i.e. it is used for preserving information, for communicating 
situations to operators and domain experts, and for knowledge representations that may 
be used in situation assessment algorithms. 
AKTiveSA (Smart et al., 2005), aims to explore knowledge-based approaches to 
information fusion and enhanced situation awareness in military operational contexts 
specifically humanitarian assistance and disaster relief. To improve SA, AKTiveSA uses 
domain ontologies that enhance information exchange and inter-operability between 
diverse agencies and user communities e.g. Non-Government Organizations (NGOs), 
United Nations (UN) agencies, coalition forces, government departments, etc. The idea 
64 
 
of semantic filters is used to provide filtered views of the situation picture to support 
task relevant information processing. Knowledge monitors actively watch the situation 
for incidents that affect current plans or problem solving goals by making use of 
semantic queries that execute as background processes. The monitors continuously 
watch the entirety of the information space and update users about relevant changes. 
 Liu & Fang (2006) developed task ontology to support modelling of activities in 
emergency response to debris-flow. Little and Rogova (2005) used ontology as a 
knowledge base for sharing and extracting information. They used ontology to provide a 
contextual understanding of the post disaster environment by supporting situation 
awareness of different users. Wenjun et al. (2005) uses ontologies for semantic 
annotation of emergency events to match the current emergency situations with past 
ones. Hoogendoorn et al. (2005) used ontology to formally specify, analyse, and 
compare disaster plans. 
All of the ontology based systems to improve SA discussed above focus on individual 
SA. However, this thesis extends the use of ontologies for automatic SA assessment to 
support SA, SSA, and TSA of emergency managers. One of the most important reasons 
to use ontology for SA modelling can be automatic assessment of a situation. Ontology 
is machine readable so a system that supports ontology modelled situations 
automatically understands changes in the situation by integrating incoming information 
on the fly. In addition, the system can run on an inference engine to compare possible 
risky evolutions of a situation. Therefore, the whole process of situation assessment, 
which results in SA, can, in principle, be automated to support all levels of human SA 
including shared and team situation awareness. 
2.8 Summary and Conclusions 
Emergency management is the planning for and the execution of all emergency 
functions necessary to mitigate, get ready for, respond to and recover from emergencies 
and disasters. The mass evacuation of an area is needed when a natural or technological 
hazard threatens and risks the safety of those within the area. Mass evacuation can also 
be needed following the impact of a hazard, which has subsequently made the area 
unliveable. A timely evacuation decision and successful in time evacuation of people 
from an area at risk can save many human lives. Making a right and timely evacuation 
decision is based on the correct interpretation of incoming situation information.  
65 
 
This chapter presents an overview of the current literature on situation awareness and 
its relationship to the design and use of emergency management systems. It synthesizes 
various concepts related to emergency management and discusses the factors that 
influence decision making in emergency situations The emphasis is on systems for mass 
evacuation but the concepts and outcomes transfer across the whole spectrum of the 
use of information systems for decision support in emergency responses. 
The literature reveals broad agreement on the distinguishing features of systems that 
support emergency response and the dynamic, stressful and time limited environment in 
which managers have to make decisions. Researchers also concur on the importance of 
the role of situation awareness (SA) in decision making but there is no comprehensive 
understanding of the way in which shared and team situation awareness operate in 
emergency teams. This is apparent both in the range of terms and models used to 
describe SA and in the focus on system designs that emphasise only individual SA. More 
work is needed on the cognitive aspects of shared awareness. Endsley?s seminal three-
level model of SA offers a good basis for such studies. 
The extension of systems design to target the enhancement of shared and team 
awareness is essential for the future development of emergency management 
information systems. An encouraging theme emerging in several research groups is the 
application of ontologies to systems design. Coupled with semantic web technologies, 
ontological design offers several advantages, not the least of which is the prospect of 
applying the SA information generated in an emergency to inference projections of 
future situation states.  
To date much of the research on situation awareness in emergency management systems 
has concentrated on individual situation awareness. More evaluation studies are needed 
to determine which investigative concepts and directions are likely to progress beyond 
the exemplar stage. Similarly, it would be fruitful to collect data from using simulation 
experiments similar to real life incidents to assess the value of the applied models. 
 
 
 
66 
 
 
 
 
 
 
 
 
 
 
 
Figure 2.8 Summary of literature review and theoretical framework  
 
 
 
 
 
 
  
 
 
 
 
Emergency Management 
Emergency Decision-making  
 
Situation Awareness (SA) 
 Shared SA (SSA) & Team SA (TSA) 
Ontology-based 
SA for IS design 
Existing Information Systems  
Information system 
support for SSA and 
TSA 
SA oriented system design (SAOD) 
SSA and TSA 
oriented system 
design 
67 
 
 
RESEARCH METHODOLOGY 
3.1 Introduction 
This chapter provides an overall research design and the methodological overview of 
the study in hand. It starts with discussing the philosophical grounding of the Design 
Science Research (DSR) and then it explains the rationale of choosing the DSR. The 
chapter then discusses the cycles and phases of the DSR in detail. While explaining the 
phases, the chapter summarises the identification of problems and opportunities in an 
emergency decision making environment with special reference to mass evacuation, 
suggested solutions, design, development and evaluation of the artefacts.  
3.2 The Philosophical Grounding of Design Science Research 
An information system, within an organization supports the processes and functions of 
the organization to improve its effectiveness and efficiency. Thus, the usefulness of the 
information system is dependent on the organization?s characteristics e.g., its work 
structure, its human resources and most importantly, the design and development of the 
information system (Vaishnavi & Kuechler, 2008). Therefore, the information system 
researchers are working to generate the knowledge that improves the design, 
development and management of the systems (Zmud, 1997).  
Such knowledge is acquired via two complementary but distinct paradigms, behavioural 
sciences, and design sciences (Hevner et al., 2004; March & Smith, 1995). Behavioural science 
originates from the natural science paradigm and seeks the truth. It involves hypothesis 
generation, data collection, and either approval or disapproval of the hypothesis. As a 
result a theory develops. Such theories help researchers and practitioners to understand 
the interactions between humans, technology and organizations. This ultimately affects 
the design decisions and results in better design, development and management of 
information systems. 
In contrast, design science is a problem solving paradigm whose goal is to produce an 
artefact, by using analysis, design, implementation and evaluation. Working with the 
technology, going through the design and development of artefact and understanding 
68 
 
the issues with the usage of artefact, is at the core of this research paradigm. The 
knowledge generated by this research informs us how an artefact can be improved and 
how it is better than the existing solutions. Knowledge created as a result of DSR can be 
in the following forms (March & Smith 1995): 
 Constructs (vocabulary and symbols) 
 Models (abstractions and representations) 
 Methods (algorithms and practices) 
 Instantiations (implemented and prototype systems) 
 Better design theories 
DSR in IS design process, provides solutions to what are considered to be the ?wicked? 
problems (Brooks 1987; Rittel & Webber 1984). Cases with such problems have the 
following characteristics: 
 Ill-defined environments; where user requirements are unclear and unstable   
 Complex problems, i.e. problems with many intermingling sub-problems  
 Design processes flexible to changes 
 Effective solution depends upon the human cognitive abilities, e.g. intelligence 
or creativity 
 Effective solution depends upon the human social abilities (e.g., teamwork) 
All of the above problems are evident in the design of the information system to be 
used for decision making and planning in emergency situations.  Therefore, current 
study uses the DSR approach to achieve its research objectives.  
In philosophy, ontology is the study about the nature of being, existence, or reality 
(Gruber, 1993). Whereas, epistemology is the study that explores the nature of 
knowledge, e.g. what is knowledge, how is it acquired, how we know what we know, 
how much can be known etc. Moreover, axiology is the study of values e.g., what values 
a certain group of people hold. The ontological and epistemological research 
approaches, which are implicit in natural and social science research, have been 
described in a number of studies (Bunge, 1984; Guba & Lincoln, 1994). Whereas, Gregg 
et al. (2001) added the design science research approach to the existing positivist and 
interpretive research approaches (Vaishnavi & Kuechler, 2008). The philosophical 
perspective of the design science researcher changes as research progresses through the 
69 
 
phases of DSR. In information system prospective, when a design science researcher 
identifies a problem in a system, he/she creates an IS (reality) assuming some hypothesis 
depending on the predictions (theory), to solve the problem. The researcher then 
reflectively becomes a positivist observer, recording the behaviour of the system, and 
comparing it to the predictions (theory) set out during the initial phases. The analysed 
observations become the basis for the new theories and a new research cycle begins.  
3.3 Rationale for Choosing the Design Science Research  
Design science research helps the researchers to understand the interactions of the 
human, the information system and the technology. Moreover, it has the potential to 
provide deep understanding into the phenomenon including the design, development, 
and management of the information system. This study intends to answer the questions: 
What are the information requirements of emergency managers to develop Situation 
Awareness (SA), Shared Situation Awareness (SSA), and Team Situation Awareness 
(TSA) during a mass evacuation operation?  How can ontology be used to support SA, 
SSA, and TSA? Whether SA, SSA, and TSA oriented system design of SAVER can 
improve SA, SSA and TSA of emergency managers during mass evacuation operations?   
All these research questions justify the use of an approach that allows some features of 
interpretive study, which is needed to understand the requirements of the emergency 
managers. For example, the approach provides the design and development of an 
artefact using ontologies to see how an ontology-based system can be developed to 
support SA, SSA and TSA. In addition, it allows the evaluation of the artefacts that may 
have helped to improve the SA, SSA and TSA of emergency managers. All these 
features are present in a DSR approach.  
Since the research questions focus on the complexity of human sense making of an 
emergency situation, the tasks and requirements (physical and cognitive) are dynamic 
due to the evolving situation which falls under the category of ?wicked? problems as 
discussed above in section 3.2. Consequently, DSR becomes an ideal approach to meet 
the challenges of such complicated problems.  
Moreover, this study uses people?s interpretations to understand their information 
requirements and the steps involved in comprehending a situation. Moreover, to keep 
the study legitimate and clear, the method used for the requirements gathering is 
discussed in detail. Therefore, the requirement elicitation part of this study can be 
70 
 
categorised as interpretive epistemology within a DSR methodology. Since, the research 
question about the user requirements is open ended and non-directional, a mix of 
qualitative and quantitative approaches is used.  
3.4 Design Science Research Cycles 
Design Science Research goes beyond the idea of innovative design. In this study, DSR 
results in clear contributions to the knowledge base and takes several forms including 
constructs, models, methods and instantiations (March & Smith 1995). However, the 
important and crucial understanding of emergency management is achieved by 
recognizing the existence of three DSR cycles in the project as shown in Figure. 3.1.  
           Environment       Design Science Research             Knowledge base  
 
 
 
 
 
 
 
 
Figure 3.1 Design Science Research Cycles (inspired by Hevner, 2007) 
3.4.1 The relevance cycle 
DSR is driven by the need to improve the system by proposing novel artefacts or the 
methods for developing these artefacts (Simon, 1996). DSR usually starts with the 
identification and representation of opportunities and problems in an actual application 
domain. Therefore, in this study, the relevance cycle initiates DSR with an application 
context that helps in describing the objectives of the research (e.g., the 
opportunity/problem to be addressed) and outlines the acceptance criteria for the final 
evaluation of the proposed artefact or research results.  
Application domain: 
Group Emergency 
operation centres 
(GEOC) 
Roles: 
Planning and intelligence 
Operations 
Welfare 
Logistics 
Controller 
Problems: 
Short available time 
High stress level 
Dynamic needs 
Evolving situation 
Diverse goals and tasks 
Team co-ordination  
Foundations: 
Design & Development: 
Decision theories  
Rational decision model 
Naturalistic decision 
model 
Situation awareness (SA) 
Team SA  
Shared SA 
SA oriented design 
Ontology-based SA 
Goal directed information 
analysis (GDIA) 
Evaluation: 
Situation awareness global 
assessment technique 
(SAGAT) 
Design           
and 
Development 
 
Evaluation 
Design 
Cycle Rigour 
Cycle 
Relevance 
Cycle 
71 
 
Decisions are the most important part of emergency management activities. To make 
correct and timely decisions, a certain level of SA, SSA and TSA is required by 
emergency managers. Current study is focused on supporting and improving the SA, 
SSA and TSA of emergency managers using a computer based system, so that their 
decision making and task performance capability can be improved. The design and 
development of such a computer based system is a vital part of this research.  
Evaluation of the artefacts helps in answering questions like; does the artefact solve the 
problem or improve the environment and how can this be measured? (Hevner et al., 
2004). The field study of the artefact can be done using various empirical methods and 
techniques similar to theory testing (March and Smith 1995), action research (Cole et al. 
2005; Jarvinen 2007), controlled experiments, simulation or scenarios. The results of the 
field testing determine whether any further iterations of the relevance cycle are needed 
in this DSR project or not (Vaishnavi & Kuechler, 2008).  
Evaluation of the artefact reports problems in the functionality, performance or 
usability that may bound the usefulness of the artefact in its application domain. Field 
testing may also show that the identified requirements were incorrect or incomplete so 
that the proposed artefact is satisfying the requirements but is still inadequate to solve 
the problem. Consequently, an additional iteration of the relevance cycle may be 
required while considering the received feedback from field testing.  
An evaluation criterion for the proposed design is based on whether the design activities 
successfully used the opportunities arising from the application of semantic web 
technologies or not. The evaluation must make sure that the requirements gathered are 
comprehensive enough to develop the required levels of SA, SSA and TSA during a 
mass evacuation operation. In addition, the tests have to check if the proposed system 
improves the SA, SSA and TSA of the emergency manager during the mass evacuation 
operations, which is the main aim of the study.   
3.4.2 The rigour cycle 
A huge knowledge base of scientific theories and engineering methods provide the basis 
for rigorous DSR. The knowledge base also contains two additional types of knowledge 
in that application domain (Vaishnavi & Keuchler, 2008): 
 The experiences and expertise that is the state of the art 
72 
 
 The existing artefacts and processes to develop them  
The rigor cycle, bases the research project on the past knowledge to ensure its novelty. 
The research thoroughly references the existing knowledge base to guarantee that the 
new design produced is a research contribution and not a replica of existing artefacts or 
the methods to develop them. While applying the existing knowledge base theories to 
the research project, care is taken to make sure that the sources and types of rigor are 
appropriate for the DSR project in hand; this will also help in providing novelty to the 
new design.  
The theories used in this research include decision theories (Klein, 1999), which suggest 
situation awareness (SA) as the most important factor that is responsible for rapid and 
effective decision making in emergency situations. In addition, since a team of decision 
makers is involved in large scale emergencies like mass evacuation as a result of tsunami 
or volcanic eruption threat to a large city, therefore apart from an individual?s  SA, 
having an acceptable level of shared and team SA (SSA and TSA) is also important.  
Moreover, an SA oriented system design (Endsley et al., , 2003a) can be used to design 
and develop the system to support SA and the resultant decision making of emergency 
managers. In the current study this approach is used for designing and developing an IS 
for supporting the SA, SSA and TSA of an individual and a team during a mass 
evacuation operation. To capture the dynamic information requirements needed to 
develop and maintain SA, SSA and TSA, a cognitive tasks analysis technique, Goal 
Directed Information Analysis (GDIA) (Prasanna et al., 2009) is used. GDIA has been 
used to acquire the information requirements of the different roles of emergency 
managers in a fire emergency. For the current study, this technique is extended for 
tsunami scenarios. Information requirements for developing SSA and TSA are also 
captured.   
It may not be appropriate or even feasible to base all design decisions on grounded 
behavioural or mathematical theories as such theories may as yet be undiscovered or 
incomplete and the research activities of design and evaluation of the artefact may 
advance the development and study of such theories. Current study is also expected to 
reduce the differences found in various definitions of SA, TSA, and SSA by providing a 
framework for defining them. Rigor in the DSR depends on the researcher?s selection 
73 
 
and the application of the appropriate theories and methods for the design, 
development, and evaluation of the artefact.  
In the current study, the widely accepted technique - Situation Awareness Global 
Assessment technique (SAGAT) (Endsley, 2000) is used to measure SA and evaluate the 
artefact. This technique is extended to measure SSA and TSA. In addition, inspiration 
for a creative design activity can be drawn from the application environment, existing 
artefacts and theories, to develop effective problem solving approaches (Iivari 2007). 
Results, which can be in the form of extension to the existing theories, new original 
theories or evaluation and feedback related to the artefact, are added to the existing 
knowledge base. It is apparent that this DSR project makes a persuasive case for its 
rigorous base and contributions to the knowledge base of an emergency management 
information system.  
3.4.3 The design cycle 
The design cycle is at the core of any DSR project. It iterates between the design of an 
artefact, its development and its evaluation to refine the design further. Simon (1996) 
explains this cycle as developing design alternatives and evaluating them against 
requirements until an acceptable design is achieved which fulfils the requirements. As 
described in the above sections, the relevance cycle provides the requirements as an 
input to the project and the design, development and evaluation methods and theories 
are drawn from the rigor cycle (Vaishnavi & Kuechler, 2008). However, the design cycle 
is where the actual DSR is done. During the design cycle, there must be a balance 
between the efforts spent on developing and evaluating the artefact and both activities 
must be credibly built on relevance and rigour. Having a strong grounded argument for 
the development of an artefact is not enough if its evaluation is not properly done 
(Iivari, 2007). 
In this study, the focus is on a balance between the development and evaluation of the 
artefact. The research described here provides the creativity required by the DSR at 
every step. For example, from requirements elicitation to design and development and 
evaluation at the end, most appropriate methods and techniques (as discussed in the 
previous section) are chosen from the relevant literature and are used creatively to meet 
the needs of the project in hand. Moreover, ontology based SA is suggested to meet the 
74 
 
requirements of the dynamic information integration and situation modelling to support 
SA, SSA and TSA, which is a major innovation in itself.  
3.5 The Design Science Research Methodology (DSRM) 
In this section, the general design science research methodology (DSRM) is described 
with reference to the research in hand. Peffers et al. (2008) propose a DSRM for 
information systems. This work provides a commonly accepted framework for carrying 
out a successful DSR project in an information system, which avoids more ad hoc 
approaches (Peffers et al., 2008; Vaishnavi and Kuechler, 2008; Hevner and Chatterjee, 
2010). Figure 3.2 shows, a typical framework for design science research.  
 
Figure 3.2 General design cycle (adopted from Vaishnavi and Kuechler (2008)) 
In the current study, design begins with the awareness of the problem. From this stage, 
the problem is identified and defined. The next stage suggests, preliminary solutions that 
are abductively drawn from existing theories and practices rooted with the emergency 
decision making domain. In the next stage, design decisions are made, and a prototype 
solution based on information requirements is developed. After completion, the 
prototype is evaluated in the final stage according to the stated objectives and criteria. 
These stages are now described below in detail.   
3.5.1 Awareness of problem/opportunity 
This step is also called ?Problem identification? (Hevner and Chatterjee, 2010) or 
?Opportunity identification? referring to the problem of improving the existing system 
or environment. In this step, the research problem or opportunity is identified and 
75 
 
described. An awareness of the problem or opportunity can come from various sources; 
for example, from an existing problem that was previously ignored or from a new 
problem generated by new developments. Similarly, the opportunities can appear with 
the development of new technologies to improve the performance of the staff or 
processes used in an organization. The output of this phase is a proposal for a new 
research effort to solve the problem or realise the opportunity. 
In the current study, the awareness of the problem came from the national level 
exercises (namely; Ruaumoko and Tangaroa) performed in New Zealand to test the 
nationwide response of organizations to large scale emergencies like volcanic eruption 
or tsunami risk. During the exercises, it was noticed that huge amounts of time were 
consumed in communicating about the situation information (sitreps) and co-ordinating 
activities and in dealing with information overload (Ruaumoko final report, 2010; 
Tangaroa final report, 2011; Javed et al., 2011b). Therefore, a computer based 
information system is needed, which can help emergency managers in developing and 
maintaining SA and the overall situation picture. Further, such a system is also needed 
to support co-ordination of activities among a team. 
To design and develop an artefact that can effectively address the problem definition or 
opportunity, it may be helpful to break the overall problem into smaller simpler and 
specific problems so that the solution can capture all aspects of the problem and 
nothing is left out (Hevner and Chatterjee, 2010). Therefore, the problem related to the 
design and development of the prototype system to support SA, SSA and TSA of 
emergency managers is decomposed into three sub-problems and discussed individually 
in the upcoming chapters. However, the summary of the problem division is given 
below and shown in Figure 3.3: 
 System design using an ontology based SA modelling (details in chapter 4) 
 System design supporting individual SA (details in chapter 5) 
 System design supporting shared SA and team SA (details in chapter 6) 
 
 
 
76 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 3.3 Dividing the problem in DSRM according to the thesis chapters 
With the recent spread of semantic web systems, an ontology based SA modelling can 
be considered as an opportunity, which can lead to many advantages. Moreover, it can 
also be a solution to some of the problems of system design and development in the 
emergency management domain. Detail of all these applications and advantages of 
ontology based SA modelling are left to Chapter 4 to avoid repetition.  
3.5.2 Suggested solution to the problem  
This activity is also called ?Define the objectives for a solution? (Hevner & Chatterjee, 2010). 
In this phase, a proposed solution is explained with all of its objectives, benefits, and the 
Problem 
identification 
Suggested 
solution 
Design & 
Development 
Evaluation 
Automatic 
situation 
assessment 
Ontology-
based SA 
oriented 
system design 
Design and 
development 
of SAVER 
Testing 
SAVER 
inference in 
various 
scenarios  
 
Chapter 
4 
SA oriented 
user interface 
(UI) and 
personalised 
information 
Ontology-
based Context 
modelling 
Context based 
personalised 
information & 
SA oriented 
interface 
Measuring SA 
& HCI 
evaluation  
 
Chapter 
5 
SSA and TSA 
support 
Ontology-
based SSA and 
TSA oriented 
System design 
Design & 
development 
of SAVER to 
support SSA 
& TSA  
Measuring 
SSA & TSA  
 
Chapter 
6 
77 
 
justification for its design and development.  This motivates the researcher and the 
stakeholders to pursue the solution, to accept the results and to understand the 
relationship between the problem and the proposed solution (Hevner & Chatterjee, 
2010). The Suggestion phase follows immediately after the Proposal. In any formal 
proposal for design science research, a proposed design and the likely performance of a 
prototype based on that design are of utmost importance. After consideration of a 
problem, if a proposed design does not appear to be promising enough to the 
researcher, the proposed idea will be modified, replaced by another one or simply 
dropped.  
Suggestion is a creative step wherein the working of a new system is projected based on 
a novel configuration of existing or new features. Moreover, in this step, the objective of 
the solution in relation to the problem definition and knowledge of what is conceivable 
and practical is described. The objectives can be quantitative, e.g., using terms to 
describe how a proposed solution would be better than the existing ones, or qualitative, 
e.g. a description of how a new artefact is anticipated to provide a solution to the 
problem. The objectives should be justified logically from the problem specification. 
Resources required for this include knowledge of the proposed solution and current 
solutions, if any, and their efficacy. 
In this research, a computer based information system is proposed as a solution to the 
problems discussed in the previous section. The system can support individual Situation 
Awareness (SA), Shared Situation Awareness (SSA), and Team Situation Awareness 
(TSA). Such a system can help emergency managers to make accurate decisions and 
perform tasks effectively and efficiently during emergency situations e.g., a tsunami 
threat to a large city. It is hypothesised that proactive sharing of information by a 
computer based information system can update human a decision maker?s SA, as any 
other team member does. In addition, making the suggestions that assist the 
comprehension of the prevailing situation and information needed to complete relevant 
tasks can reduce the time and effort needed to develop SA, SSA and TSA.  
A further objective of system design using ontology based SA modelling is to see how 
ontology can be used to model SA. Existing ontology based SA modelling techniques, 
e.g. by Matheus et al., (2004) are extended to see whether SA modelling requirements of 
the current study can be fulfilled using ontology or not. Moreover, Matheus et al., (2004) 
78 
 
only provided a top level situation ontology, which needs extension to a real scenario. 
More detail of the suggested solution along with the developed ontologies is provided in 
Chapter 4. System design supporting individual SA is achieved by an SA oriented system 
design that is a user centred design approach (Endsley et al., 2003a). In addition, 
personalised information according to the context should be presented to the 
emergency managers to avoid information overload. Further details of suggested user 
interface and contextual information presentation are covered in chapter 5. System 
design supporting shared SA and team SA depends on the correct identification of SSA 
and TSA requirements. Therefore, the SSA and TSA requirements should guide the 
design of a system, supporting SSA and TSA. Details of SSA and TSA requirements and 
the suggested solution to SSA and TSA systems design are described in chapter 6.  
3.5.3 Design and development 
As the name shows, this activity is concerned with the design and development of the 
proposed artefact. The artefacts are in the form of a model, method and a prototype 
(Hevner et al. 2004). Theoretically, a DSR artefact can be any designed entity in which a 
research contribution is embedded in the design. This activity includes designing the 
artefact architecture to provide the required functionality and to fulfil the desired 
objectives listed in the previous step and then developing the real artefact (Hevner and 
Chatterjee, 2010).  
Designing a successful artefact requires a lot of innovative exertion. Apart from using 
the existing theories and processes, innovation has to be applied in this step to make the 
artefact unique and useful among the existing solutions. Therefore, it is the most 
challenging activity in the DSRM. The completed design is transformed into the real 
artefact that embodies the features that solve the identified problem.  
Development methods again vary, depending on the artefact to be constructed and the 
design decisions to be made.  For example, an algorithm may require construction of a 
formal proof whereas an expert system expressing novel features in certain areas of 
interest may require software or tool development, which is the case in the current 
research. The implementation itself can be very ordinary and does not need novelty 
beyond the normal development practice for the similar artefact thus the  novelty is 
mainly in the design, not the construction of the artefact (Hevner and Chatterjee, 2010; 
Vaisnavi and Kuechler, 2008). Current research has the clear objective to design a 
79 
 
computer based system to improve SA, SSA and TSA of emergency managers in an 
emergency situation. Relevant theories, methods, and techniques are creatively applied 
in design and development. These design and development decisions are discussed 
below. 
3.5.3.1 Design decisions 
To provide computer based SA support, situation information needs to be modelled in 
detail. However, the first step in this regard would be to identify the information 
required to develop and maintain SA in a particular scenario. As discussed earlier, along 
with the physical tasks, many cognitive processes are also involved in SA development; a 
cognitive task analysis technique is needed to identify these information needs. 
Extended Goal Directed Information Analysis (EGDIA) is used for information 
elicitations of various user groups. Details can be found in chapter 4.  The second step 
would be to provide the information needed for developing and maintaining a required 
level of SA by using ontologies for:  
 Automatic information integration  
 Modelling situation evolution  
 Classifying information according to the different levels of SA 
Moreover, ontologies can also solve the problem of the dynamic information 
requirements of emergency managers in evolving emergency situations, as ontologies are 
easy to update dynamically and are very flexible to model situation (Matheus et al., 2004) 
and context (Chen, 2004) information. In addition, ontologies, when used with a 
reasoned, also provide inference capability to show how a situation may evolve. More 
details of ontology  based design are discussed in Chapter 4.  
The system to support SA is devised by following the SA oriented design process 
(Bolstad et al., 2006; Endsley, 1995b). This design process suggests developing user 
interfaces, which provide goal and task oriented information. This design method also 
promises that providing the higher level of SA information to the user, improves 
his/her SA instead of simply providing the raw data which needs further processing. 
Moreover, the context information is used to provide the personalised and specific 
information to emergency managers during a mass evacuation operation to minimise the 
problem of information overload. Details of these design decisions are covered in 
chapter 5.  
80 
 
To deal with the problem of system design for supporting SSA and TSA, SA oriented 
design is extended to provide higher levels of Shared and Team Situation Awareness 
(SSA and TSA). Hence, information requirements needed for SSA and TSA are 
identified by keeping the individual SA requirements and various definitions of SSA and 
TSA in view. Further details of system design to support SSA and TSA are provided in 
chapter 6.  The prototype designed to address these issue is dubbed Situation Aware 
Vigilant Emergency Reasoner (SAVER) and its overall design steps are shown below in 
Figure 3.4.  
 
 
 
 
 
 
 Figure 3.4 Steps of design 
3.5.3.2 Development decisions for ontology based SAVER   
Web Ontology Language (OWL6) is used for developing the ontologies needed for the 
implementation of SAVER. World Wide Web (W3C) endorses OWL for authoring 
ontologies. OWL is characterised by the formal semantics and uses serialisation, based 
on Resource Description framework (RDF7) and Extensible Mark-up Language (XML). 
The reason for choosing OWL is that it is a modelling language that can facilitate 
greater machine interpretability of Web content rather than that supported by XML, 
RDF and RDF Schema (RDF-S). It is because OWL provides an additional vocabulary 
along with the formal semantics.  
                                                 
 
6 http://www.w3.org/TR/owl-features/ 
7 http://www.w3.org/RDF/ 
Information requirements EGDIA 
Situation Awareness 
Shared and Team Situation 
Awareness
Ontology support 
User Interface Design SAOD
81 
 
Semantic Web Rule Language (SWRL) 8  is used to state the rules for defining 
relationships between various domain concepts and applying restrictions to these 
relationships. SWRL is a semantic web rules-language, combining OWL with those of 
the Rule Mark-up Language (unary/binary data-log). SWRL has the full power of OWL 
DL (description logics).  The domain and application ontologies were developed using 
Prot?g? 4.0.29 and the programming interface to ontologies were developed in Java 
using the OWL application-programming interface (API)10 and Jena11 API. OWL API is 
an open source reference implementation for creating, manipulating, and 
serialising OWL ontologies and is available under the Lesser General Public license 
(LGPL) 12 . Jena is a Java framework for building semantic web applications. Jena 
provides a collection of tools and Java libraries to develop semantic web and linked-
data applications, tools and servers. Fact Plus Plus (Fact++)13 is used as a reasoner for 
ontology validity and inference. A reasoner is a key component for working with OWL 
ontologies. Virtually all querying of OWL ontology is done using a reasoner. This is 
because knowledge in ontology might not be explicit and a reasoner is required to 
deduce implicit knowledge so that the correct query results are obtained. The OWL API 
includes various interfaces for accessing OWL reasoners.  
The front end/user interface was designed using the Vaadin14 application framework. 
Vaadin is a web application framework for Rich Internet Applications (RIA). Unlike 
JavaScript libraries and browser-plugin based solutions, it features robust server-side 
architecture. Therefore, the largest part of the application logic runs securely on the 
server. Google Web Toolkit (GWT)15 is used on the browser side to ensure a rich and 
fluent user experience. The main reason for choosing Vaadin is its easy programming 
model. Moreover, Vaadin is a robust architecture for rapid application development. 
The component based architecture together with statically typed Java language and data 
binding features helps in building applications that are easily modularised and refactored 
                                                 
 
8 http://www.w3.org/Submission/SWRL/ 
9 http://protege.stanford.edu/ 
10 http://owlapi.sourceforge.net/ 
11 http://incubator.apache.org/jena/ 
12 http://www.gnu.org/licenses/lgpl.html 
13 http://owl.man.ac.uk/factplusplus/ 
14 https://vaadin.com/home 
15 http://code.google.com/webtoolkit/ 
82 
 
as needed. The integrated development environment (IDE) and support including visual 
designing tool helps to build a web user interface extremely fast. The system 
architecture showing components of SAVER is shown in Figure 3.5 below: 
 
 
 
 
 
Figure 3.5 System architecture of SAVER 
3.5.4 Evaluation  
Once developed, the artefact must be evaluated against the problem definition that is 
implicit and frequently made explicit in the section 3.4.1 ?Awareness of problem? and 
the objectives set in section 3.4.2 under the heading ?Suggested solution to the 
problem?. Deviations from expectations, both quantitative and qualitative, are carefully 
noted and must be justified. The evaluation phase contains a sub-phase in which 
hypotheses are prepared about the behaviour of the artefact based on objectives set in 
suggested solution and problem identification phases. For example, in current research, 
it is hypothesised that SA oriented design will support SA of emergency managers and 
the proactive sharing of information according to SSA and TSA requirements will 
improve SSA and TSA of emergency managers. Additionally, in the other sub-phase, an 
analysis to either confirm or contradict an hypothesis is provided. In conventional 
research, after discussing the considerations for future work, the research effort is 
finished but for the design science researcher, by contrast, it is rare that a hypothesis is 
completely borne out after the first cycle of DSR.  Instead, the evaluation phase results 
and knowledge gained during the design and development phase is fed back to 
Suggestion (cf. see the arrows going back in Figure 3.2) for another cycle. This concept 
has been observed by philosophers of science in many areas (Lakatos, 1978); and 
working from it, Newell suggests that theories should be treated like doctoral students 
who are corrected when they make a mistake and they go on to be ever more valuable 
and creative (Newell, 1990).  
OWL API / Jena API 
OWL (Ontologies) SWRL (Rule 
Language) 
FACT++ 
(Reasoner) 
Java Field 
Data 
Vaadin  
(User Interface) 
83 
 
Evaluation of the artefact involves its use in experimentation, simulation, case study, 
proof or any other suitable and related activity. Performance of the artefact in the field 
is observed and measured to see how well the artefact supports a solution to the 
problem. This phase requires knowledge of relevant metrics and analysis techniques. 
Evaluation can be of many types depending on the problem domain and features of the 
artefact itself. The artefact?s functionality and the testing results are compared with the 
solution objectives from the ?Suggestion? phase described above. Evaluation may also 
include objective quantitative performance measures such as the number of items 
produced, quantifiable measures of system performance such as response time, 
availability or increase in productivity against used resources (efficiency), satisfaction 
surveys or client feedback. Therefore, the evaluation could include any suitable empirical 
proof or logical evidence. At the end of evaluation, the researcher decides whether it is 
required to iterate back to the previous steps to improve the effectiveness of the artefact 
or to leave further improvements for subsequent projects (Hevner & Chatterjee, 2010). 
The nature of the DSR and application domain may also direct whether iteration is 
possible or not. 
The evaluation of the design in the current study starts with the testing of ontology-
based modelling of SA, i.e. to examine if ontology can be used to model a situation. This 
is done by testing the system for inference capabilities using test cases, which are 
prepared as scenarios for evaluation to assess if the inference and suggestions made by 
the system are correct and useful. The details of testing process, scenarios, and results 
are provided in Chapter 4.  
The proposed design in this research intends to improve SA, SSA and TSA. Therefore, 
SA, SSA and TSA are measured to evaluate the design and prototype. A widely accepted 
technique, Situation Awareness Goal Assessment Technique (SAGAT) (Endsley, 
1995a), is used to measure individual SA.  Since user interface design plays an important 
role in improving SA, it is also evaluated using human computer interaction (HCI) 
matrices (Bailey et al., 2000; Sauro & Kindlund, 2005; Tullis & Albert, 2008). Details of 
SA measurement and user interface evaluation along with the results are covered in 
Chapter 5. SSA and TSA are measured in experiments conducted with eight pairs of 
emergency managers. SAGAT is extended for measuring SSA and TSA. Details of SSA 
and TSA measurements are described in Chapter 6. Figure 3.6 summarizes the principle 
of its evaluation process. 
84 
 
 
 
 
 
 
 
 
Figure 3.6 Evaluation method 
3.5.5 Conclusion 
This is the final phase of a particular design science research (DSR) effort. This phase 
comes once satisfactory results of evaluation have been achieved and no more iteration 
is planned in the current project. Even if small discrepancies are still present in the 
behaviour of the artefact but results are good enough to justify the research effort 
within the scope of that project, research is concluded (Hevner & Chatterjee, 2010; 
Vaishnavi and Kuechler, 2008). Apart from the results, the experience and learning at 
every phase of the project is also part of the new knowledge. Abnormal behaviour that 
is confusing and cannot be explained may well serve as the subject of future research. 
New knowledge generated is reported and published so that other researchers can make 
use of it in future DSR.  
Results from the evaluation of each design are discussed in the conclusion sections of 
their respective chapters i.e. 4, 5, and 6. These sections add knowledge and learning 
from the new design process. In addition to the theory and practices, the impact of the 
new results and techniques learnt are also discussed in chapter 7. The overall DSRM, 
adopted for the current study is shown in Figure 3.7 below: 
 
 
 
Testing SAVER 
Measuring Individual SA 
(SAGAT) / HCI 
Measuring SSA and TSA 
(extended SAGAT) 
85 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 3.7 Overview of research methodology  
 
 
Evaluation 
Conclusion & Dissemination  
Testing SAVER  
Measuring individual? SA / HCI  
Measuring team?s Shared and 
Team Situation Awareness   
Research publications  
Design & Development 
 
Extended Goal Directed 
Information Analysis (EGDIA) 
Ontology-based Situation 
Awareness modelling   
SAVER (prototype) development   
Simulations & experiments  
Problem / opportunity 
identification 
Suggested Solution to the 
Problem 
National Exercises reports, 
observations, & literature  
Existing decision-making 
theories and literature 
Information requirements  
Ontologies 
SAVER (prototype) 
Ontology-based SSA, TSA  
86 
 
 
SAVER: SITUATION AWARE VIGILANT 
EMERGENCY REASONER 
4.1 Introduction  
Decision making is a vital part of emergency management. In every phase of an 
emergency, decisions are made to control the situation or return it to normal. To make 
these decisions mangers need to integrate different types of information from diverse 
sources so they can analyse and interpret the material to form an accurate understanding 
of the situation.  
Giving meaning to the incoming information is a complex task as many factors, e.g. 
attention level, past experience, cultural background, training, intelligence etc. can 
influence the process. Consequently, different people can generate very different 
interpretations of the same incoming information.  
This research proposes a computer based system to support the information analysis of 
managers so they can collectively form a more realistic understanding of the emergency 
and deliver optimal decisions. Semantic web technologies are proposed to model the 
domain knowledge of the emergency situation and automatic information analysis to set 
the basis for improved situation awareness.  
This chapter first elaborates the problem relating to the requirement of computer based 
situation awareness (SA) and opportunities arising from having automatic information 
integration for SA. Moreover, a problem of acquiring the detailed SA requirements is 
discussed. In the next section, a cognitive tasks analysis technique is suggested for 
requirements elicitation. In addition to this, the design and development of a computer 
based system to enhance SA is explained. The next section, explains the design and 
development of the proposed solution prototype. The chapter then presents the 
evaluation of the design and prototype for successful situation modelling and accurate 
and useful information analysis by using four different simulated scenarios. The section 
on evaluation of the design and prototype is followed by a conclusion section.  
87 
 
4.2 Problem Identification  
For emergency decision making during a mass evacuation operation, there is a need to 
receive or acquire the relevant information from diverse sources, integrate this 
information and present it to decision makers according to their requirements in a 
readily usable format. Doing all of this work manually takes a lot of time and can result 
in delayed decisions with potentially serious consequences.  
Consequently, a computer based system is required to support human reasoning and the 
related processes that improve SA. For the successful design of such a system, human 
cognitive requirements must be considered in addition to the physical task requirements. 
However, realising these requirements is not an easy job as many cognitive processes are 
involved in developing and maintaining situation awareness. In addition, an emergency 
situation is continuously evolving and hence the requirements are dynamic according to 
the nature and phase of emergency. Realising the requirements in sufficient detail is a 
challenge. Systems designed based on these requirements must be able to integrate 
information automatically so that the incoming data can be made readily useable. This 
information interpretation requires semantic information analysis to understand the 
meaning of information in the prevailing situation.  
A computer based system that is capable of situation analysis requires a situation to be 
modelled in detail. This means all the important relationships between the concepts 
related to the domain must be defined. Nevertheless, since emergency situations are 
dynamic, i.e. continuously evolving, a situation sometimes arises that is not supported 
by the existing schema and new relations or even new tables in the databases are needed 
depending upon the situation. Therefore, runtime changes in the schema are needed. 
These are difficult to manage even using very advanced database management systems. 
The main problems can be summarised as follows: 
1. Acquiring dynamic information needed for developing and maintaining 
situation awareness   
2. Classifying information requirements according to the nature and phase of a 
mass evacuation operation  
3. Automatic integration of information from multiple sources in a 
semantically coherent and operationally useful manner  
88 
 
4. Providing information at all three levels of SA required for decision making 
and tasks completion 
4.3 Suggested Solution to the Problem 
This study suggests a computer based SA system as a solution to the problems 
mentioned above. A system capable of acquiring the required information from 
different sources, integrating it and transforming it into a readily usable form, can 
address the problems identified above. This processed information can be provided to 
the decision makers according to the requirements of their roles and goals during a mass 
evacuation operation. Moreover, a system that can interpret the meaning of incoming 
information and provide advice on all three levels of SA would be highly useful. Hence, 
the system should first have a capability to identify the relevant information on the 
perception level of SA according to the role and phase of the emergency situation. The 
system should then provide information for the comprehension (second) level of SA to 
explain the significance of the perceived information in the context of the prevailing 
situation and the user?s (emergency manager) goal. To help emergency managers make 
decisions and perform tasks proactively, the system should also predict the future states 
of the situation, i.e. deliver level three of SA information.   
Detailed SA requirements can be acquired using a technique like Cognitive Task 
Analysis (CTA). CTA helps researchers to identify what an individual (or team of 
individuals) needs to do as cognitive processes or actions to achieve his/her goals. Since 
traditional task analysis techniques fail to identify the cognitive processes associated with 
an activity (Kirwan & Ainsworth, 1992), CTA becomes an important tool for system 
designers, where identification of cognitive demands is critical for the system 
requirements (Stanton et al., 2005; Clark et al., 2008). Several CTA tools are available 
(Flanagan, 1954; Klein, 1989; Militello & Hutton, 1998; Vicente, 1999; Endsley et al., 
2003b) so an appropriate technique must be selected to acquire comprehensive SA 
requirements.   
In this research, to tackle the issue of dynamic situation modelling, ontology based 
domain modelling is used since ontology based systems can solve the problem of 
information integration from diverse sources as required in emergency situations (Smart 
et al., 2005). Other problems that can be addressed by the use of ontology based 
systems include use of semantic web crawlers for automatic information gathering, 
89 
 
dynamic situation and context modelling. In addition, the problem of dynamic changes 
in the database schema can also be handled using ontology based systems. Detailed 
advantages of a semantic web based system are described in the upcoming design 
section.   
4.4 Design and Development  
This research focusses on the design of an information system supporting the situation 
awareness (SA) of emergency managers in mass evacuation operations, specifically in 
tsunami scenarios. Design and development of a system usually starts with the 
requirements gathering process.  
4.4.1 System requirements elicitation 
Successful design and development of a system depends highly on the correct system 
requirements elicitation as the majority of software problems are attributed to mistakes 
made at the requirements gathering phase (Jones, 1994). Several authors have identified 
problems in traditional requirements engineering techniques (Robinson & Elofson, 
2004) and proposed alternatives, e.g. soft system method (Checkland, 1999), scenario 
based approaches (Weidenhaupt et al., 1998), rational unified process (Dardenne et al., 
1991).  
Since, in emergency situations, the user environment is extremely dynamic and 
continuously changing, the system must also support dynamic user requirements. 
General software engineering techniques are not well suited to capture such 
requirements in detail and with accuracy (Endsley et al., 2003b). Moreover, as described 
in Chapter 2, User Centred Design (UCD) is capable of capturing the correct 
information requirements of users by focusing on their needs instead of advising them 
on what they should use (Jennex, 2007; Endsley et al., 2003b).  
4.4.1.1 Selecting an appropriate cognitive task analysis (CTA) technique 
SA is developed and maintained by having up to date information about the situation 
therefore SA requirements are actually various types of information.  
Existing CTA techniques include Goal Directed Tasks Analysis (GDTA) (Endsley et al., 
2003b), Critical Decision Method (CDM) (Klien et al., 1989), Cognitive Work Analysis 
(CWA) (Vicente, 1999), Applied Cognitive Task Analysis (ACTA) (Millitello & Hutton, 
1998), Goal Directed Information Analysis (GDIA) (Prasanna et al., 2009). Prasanna et 
90 
 
al. (2009) provide a detailed comparison of ACTA and GDTA. They argue that ACTA 
is only capable of providing the cognitive requirements relating to a user?s performance 
whilst GDTA is capable of providing information requirements to make better 
decisions to achieve a user?s goals. These techniques can be combined to develop the 
GDIA technique so that complete and comprehensive information requirements can be 
captured (Prasanna et al., 2009). The requirements gathering process can then start by 
asking emergency managers about their tasks in a given scenario, the reasons for the 
specified tasks are queried to understand an emergency manager?s goals and cognitive 
demands. These queries lead to the information required for carrying out these tasks. 
GDIA has been successfully used for information gathering in similar complex 
scenarios, e.g. fire emergencies (Prasanna et al., 2009; Yang et al., 2009), and hence will 
be used for this research.  
4.4.1.2 Extended Goal Directed Information Analysis 
For this study, the steps of GDIA, as shown in Figure 4.1 below, are modified to meet 
the domain requirements while maintaining its validity and objectives intact.  
 
 
 
 
 
Figure 4.1 Steps of GDIA (adopted from Prasanna et al., 2009) 
Likewise, GDIA is extended by adding a step called Simulation to validate the gathered 
requirements in an actual scenario. The adapted steps of extended GDIA (EGDIA) are 
shown in Figure 4.2 below: 
 
 
 
 
Context Discovery Scenario Building 
Task Elicitation Goal Elicitation 
Goal Validation Information Elicitation 
Goal-Decision-Information 
(GDI) Diagrams 
Validation of GDI 
Diagrams 
91 
 
 
 
 
 
 
 
 
 
 
 
Figure 4.2 Steps of EGDIA (Javed et al., 2011a) 
The order of these steps is also modified to accommodate the availability of participants 
(emergency managers). For example, the validation of tasks, goals and their related 
information requirements was done in questionnaires after the interviews as illustrated 
in Figure 4.2 above. The tasks, goals, and information requirements gathered during 
interviews were categorised according to the roles and phases of emergency managers 
and were used to develop online questionnaires. Emergency managers from all over 
New Zealand were asked to complete these questionnaires in the context of a tsunami 
scenario. They were asked to rate each of the information requirements according to the 
nature and phase of the emergency situation.  Out of 17 primary experts from all over 
New Zealand, 13 participated for each role. Details of these steps are as follows: 
1. Context discovery  
Preliminary understanding of the emergency context of mass evacuation operations is 
obtained via the following sources (Hammersley & Atkinson, 1995).  
? Face-to-face interviews with relevant emergency managers 
? Observations of real time practices in natural emergency exercises; Ruaumoko 
and Tangaroa 
National 
Exercises 
Context 
discovery 
Scenario 
development 
Roles & 
Phase 
identification 
Interviews 
Task 
identification 
Goal 
identification 
Information 
requirements 
identification 
Questionnaires 
Goal 
validation 
Task 
Validation 
Information 
requirements 
validation 
S
i
m
u
l
a
t 
i
o
n 
92 
 
? Formal and informal documentation such as procedure manuals and policy 
documents produced as part of the emergency response work such as civil 
defence emergency management (CDEM) plans (MCDEM, 2009), guide to 
mass evacuation plans (MCDEM, 2008a) etc.  
Ethical approval is obtained from the Massey University Ethics Committee regarding 
the observations and interviews during the national exercises (see appendices A, B, and 
C). Interview questions are described in Appendix D. This helped in building the basic 
knowledge required to carry out the rest of the EGDIA steps. Moreover, it also helped 
in building suitable scenarios from Exercise Ruaumoko for volcanic eruption scenarios 
and Tangaroa for tsunami scenarios, which are described in the second step of EGDIA.  
2. Scenario building 
ACTA incorporates clearly defined interview probes during its application, the lack of 
which is a major limitation in GDTA. Among such probes, "Simulation Interviews? are 
most significant. The Simulation Interview is a type of interview that is specifically based 
on the presentation of a scenario to the emergency manager. Its aim is to overcome 
limitations due to the unstructured nature of interviews and to sharpen the focus of 
participants.  Moreover, scenarios are helpful to probe mental models, thinking 
processes and assumptions of participants by improving their focus and imagination.   
Scenario: The national exercise Tangaroa is used as a guideline to develop tsunami 
scenarios by simulating the information feeds. Emergency managers use seismic and 
related data to register an underwater event (e.g. earthquake or volcanic explosion) that 
could cause a tsunami. Deep Ocean Assessment and Reporting Tsunami (DART)16 data 
is then analysed to confirm whether a tsunami has been generated or not. Once a 
tsunami has been confirmed, areas of possible devastation are marked and planning for 
possible evacuation is started. Priority zones or the areas that need special attention, for 
example schools and hospitals are marked. Similarly, decisions to evacuate, and the time 
and extent of evacuation, are decided. Planning for the safe evacuation of people, which 
includes transport, food and shelter, and medical facilities are critical considerations 
                                                 
 
16 http://nctr.pmel.noaa.gov/Dart/ 
93 
 
since wrong decisions can lead to either unnecessary disruption or increased casualties 
and economic loss. 
3. Role and Phase identification 
The following main roles/functions are identified in the Group Emergency Operation 
Centre (GEOC), from interviews, national level exercises and documentation, 
(MCDEM, 2008a; 2008b, 2009): 
? Planning and Intelligence (P&I): The P & I function is responsible for 
forecasting the incident development, anticipating likely needs, and drafting the 
Incident Action Plan. This role of the GEOC has a strategic scope and provides 
other team members at the GEOC with situation analysis as the incident 
develops through various phases of the emergency. This situation analysis is in 
the form of a report called a situation report (SitRep). 
? Operations: The Operations function enacts the Incident Action Plan, which 
makes sure that responders are as focused and aware as possible so they can 
fulfil the objectives set by the Controller. The Operation is generally responsible 
for the operational command of resources. This means allocating specific 
functions to agencies in their areas of expertise, monitoring their performance 
and providing a communication link between the responders and the other 
elements of the GEOC, especially Logistics. 
? Logistics: The Logistics function or Manager ensures the continuity of 
operations by making sure that there are sufficient resources (e.g. personnel, 
vehicles, machinery, buildings, food supplies, medical supplies, tents etc.) on site 
and within GEOCs.  
? Welfare: The Welfare function is responsible for the wellbeing of affected 
people during or after the emergency situation, especially in the case of 
evacuees. The Welfare personnel set up temporary welfare camps and provide 
accommodation, food and basic medical facilities until required. 
? Controller: The Controller is responsible for the overall incident progression 
and has overall accountability for the incident. The Controller?s responsibilities 
include: 
o Safety of staff and public  
o Overall operation of GEOC 
94 
 
o Approval of Situation Reports issued by the GEOC 
o Incident stability, i.e. the strategy that will be most effective at resolving 
the incident with economical use of resources. This strategy is issued 
through the use of the Incident Action Plans (IAPs) 
o Property conservation, which relates to the overall damage limitation 
An emergency situation involving mass evacuation is divided according to the following 
phases so that information requirements can also be contextualized according to them 
and right information can be provided at the right time.  
? Pre-Tsunami Confirmation Phase: An earthquake has been confirmed but it 
is not confirmed whether a tsunami is generated at the source or not.  
? Tsunami Confirmed Pre-Decision Phase: A tsunami has been confirmed but 
a decision has not been. Threat level is being confirmed.   
? Evacuation Decision Phase: A tsunami has been confirmed and an evacuation 
decision is being made about time and areas of evacuation.  
? Evacuation Phase: The evacuation decision has been made and people are 
being evacuated  
? Post-evacuation Phase: Evacuation has been completed and people are in 
welfare centres. 
? Return Phase: People are returning to their homes.  
 
4. Tasks elicitation validation 
Each step in the development of a scenario involves the identification of tasks 
performed by each role. Each selected emergency manager is asked to: 
? Describe the possible activities they carry out under the described scenario 
? Identify possible task groupings and sub-tasks 
? Equal opportunity is given to both experienced and novice emergency managers 
to avoid any bias and to capture complete and accurate requirements. 
Identified tasks for each role are presented in Appendix E. After collecting the task lists 
from various emergency managers, the data is analysed to develop the task structure. 
The intention of this analysis is to identify tasks so they can be used together with the 
95 
 
scenarios during the next step of the research to determine the cognitive demands of 
individual emergency managers. 
Moreover, tasks identified from one emergency manager are compared with the tasks 
from the other one in the same role to find the common ones. In addition, a task missed 
by one participant was confirmed for validity by reminding him/her about it. Further, 
his/her discussion about the task is considered to understand the value and importance 
of the task. Two main tasks identified for the Planning and Intelligence function are:  
? Developing situation report (SitRep) 
? Developing Incident Action Plan (IAP) 
These two tasks are carried forward as an example to identify the information 
requirements. 
5. Goal elicitation and validation: 
Semi structured interviews were conducted with the emergency managers to identify the 
goals and sub-goals that were expected from the situations under investigation. Initially, 
ten emergency managers were interviewed during exercise Tangaroa on two shifts. Each 
manager was an active member of a GEOC and was an expert in one of the five main 
functions, i.e. Controller, Planning and intelligence, Operations, Welfare, Logistics.  
In this exercise, tasks identified in the previous section are used as probes to unveil the 
cognitive demands or goals, i.e. why each task was selected. Specific task probes and 
scenarios are expected to control and guide the interviewing process. The goal hierarchy 
forms the basis for deriving information requirements. Thus, it is very important to 
make sure that the developed goal hierarchy is meaningful and accurate. At this level of 
the process, it was very useful to check the validity of the derived goal structures. A 
larger group of emergency managers (18 members), which included emergency 
managers from various EOCs in the area and managers previously interviewed, 
validated the goals identified two days after exercise Tangaroa on exercise debriefing 
day. The elicited goals from each member were cross checked with others for validation. 
The following steps illustrate this validation process: 
? The developed goal structure is distributed among the interviewed emergency 
managers 
96 
 
? Discussions with the previously interviewed emergency managers to revisit the 
developed goal and sub-goal structures so that doubtful elements can be 
removed  
? To represent generic goals of the domain being investigated; the results are 
validated by discussions with different emergency managers who represent other 
Emergency Operation Centres (EOCs) of the same region.  
? Modifications of the goals and sub-goal structures based on the comments and 
suggestions 
To represent generic goals of the domain being investigated; the results are validated by 
discussions with different emergency managers who represent other Emergency 
Operation Centres (EOCs) of the same region. Main goals associated with the tasks 
mentioned above are: 
? Provide an up to date analysis of the prevailing situation to decision makers 
? Provide a plan to carry out safe evacuation of the area under risk 
 
6. Information elicitation and validation 
The information required to make decisions and carry out the tasks is obtained from the 
identified goals and sub-goals. To ensure that the obtained goals, decisions, tasks and 
their information requirements are complete and accurate, validation with a larger group 
of emergency managers representing other similar organisations in and outside the region 
is done. The validation process is expected to improve the generalisation of the findings. 
In this manner, the findings of the study will become a closer representation of the 
requirements of a much wider domain, such as a country, rather than one that is limited to 
a single district or region. Later, the feedback obtained is integrated with the feedback of 
emergency managers representing outside regions. Such integrated feedback could then be 
used to make the necessary corrections to the identified information requirements. In 
addition, possible sources of information and the next destination of analysed information 
is also queried and validated to enable the modelling of information flows between 
different roles.  The step wise sequence is illustrated in Figure 4.2 above.  
 
 
97 
 
7. Validation in simulation 
The requirements analysis allowed us to develop a prototype called Situation Aware 
Vigilant Emergency Reasoner (SAVER). The style in which information is provided to 
the emergency managers greatly influences their ability to make a good decision in 
reasonable time. Information requirements acquired in previous steps provide 
information to emergency managers via SAVER in a simulated scenario. This step was 
supposed to find any missing requirements or to identify problems with the existing 
information requirements. Participants were asked to perform tasks like preparing a 
situation report. Information needed for the task was provided via SAVER. Later, 
participants were asked whether they needed more information than was presented to 
see if any information was missing.  Moreover, participants were asked if there was 
anything wrong with the content or format of information presented. The next section 
details the final validated information requirements for various roles of users over the 
different phases of a mass evacuation operation.  
4.4.1.3 Information requirements 
To validate goals, tasks and information requirements, questionnaires were circulated to 
the GEOCs all over New Zealand to get feedback from the wider community. In all the 
following tables, the first column shows the phases of emergency, the second column 
shows the information requirements of each role during the respective emergency 
phase, and the third column, ?%R?, shows the percentage of participants out of the 
total participants who mentioned the identified information requirement as relevant. 
Though there is no method available to find an accurate amount of information 
requirements needed to achieve desired levels of SA, a certain number of information 
requirements can be validated as ?appropriate to achieve the goals? by multiple domain 
experts. Moreover, initially the information requirements, which are declared as related 
by 75% or more, are included in the final requirements and the ones which are 
mentioned as irrelevant by more than 25% are discussed with the participants to know 
the reasons. In addition, after mutual understanding with the domain experts 
(emergency managers), the criteria for rejecting an information requirement by declaring 
it irrelevant is fixed at 75%, however, none of the information requirements was 
mentioned as irrelevant by 75% or more participants, therefore none was rejected. The 
three main objectives of this validation via the questionnaires are: 
98 
 
1. To discover any problems with the roles, goals, tasks, and their corresponding 
information requirements  
2. To add (if any) role, task, goal, or information requirements that are missed out  
3. To determine what the wider community of emergency managers all around 
New Zealand think about the roles, tasks, goals and information requirements 
relevant to the phases of emergencies and the roles of emergency managers 
See Table 4.1 for information requirements of the planning and intelligence (P&I) role.  
Table 4.1 Planning and Intelligence (P&I) function information requirements. 
Emergency 
phase 
Information requirements (Planning & Intelligence) 
%   R 
Pre-Tsunami 
Confirmation 
phase 
Detail on the magnitude, location, time of earthquake (tsunami 
source)  
100 
The analysis of earthquake/tsunami information is required 100 
Deep-Ocean Assessment and Reporting Tsunami (DART) 
Data from National Oceanic and Atmospheric Administration 
(NOAA). 
100 
Tsunami 
confirmed-Pre 
Decision phase 
Information required for activating the GEOC  100 
The contact details of staff are required 100 
Direction, speed and height of waves from various DART 
sensors in the ocean along with the surrounding geographical 
data are required 
92 
Information will be needed as to whether there is a way to 
stop the water reaching populated areas, e.g. using sandbags 
etc. 
92 
To develop a situation report, P&I requires information on: 
US Geological Survey report on earthquake and its analysis 100 
DART data 100 
NCMC advice or warnings about the current status 100 
Situation reports from the LEOCs 100 
99 
 
Miscellaneous information, e.g. maps of areas at risk, 
declaration of national warnings, media releases 
100 
Evacuation 
decision phase 
To develop an Incident Action Plan (IAP), P&I requires: 
Area to be evacuated 100 
Population of the area 100 
Number of facilities in the area which require special attention 
e.g. schools and hospitals etc. 
100 
Number of self-evacuees till now 100 
People who will need assistance 100 
Nearest safe zones to evacuate 100 
 
Evacuation 
phase 
The current figure of total evacuees 92 
The safety of the evacuation routes 100 
The current status of DART data 
50 
Post evacuation 
phase 
The water levels in the affected area 92 
Future risk of rise in water levels 100 
Welfare conditions of evacuee camps 92 
Which other locations need attention 83 
Return Phase 
The condition of evacuee accommodation 69 
Number of self-returning evacuees 61 
Evacuees needing help for return 100 
The resource requirements of evacuees for return 100 
 
Table 4.1 summarizes the responses about the validation of information requirements 
for the Planning and Intelligence (P&I) role. Obviously, P&I being central to 
information gathering, processing, planning and dissemination has the highest number 
of information requirements. All information requirements of the ?Pre-Tsunami 
100 
 
confirmation phase? are relevant, as a tsunami is only generated if the magnitude of 
earthquake is high enough and the earthquake is in water or very close to coastal areas. 
During the second phase ?Tsunami confirmed pre-decision phase?, seven out of nine 
information requirements are identified as relevant by 100% of participants as 
information such as direction, speed and height of incoming waves help to identify areas 
at risk and the likely arrival time of waves, essential to decide the time of evacuation. 
With three exceptions, all the information requirements gathered for all remaining 
phases are marked as relevant. ?Updated DART data? for ?Evacuation phase? is 
mentioned as not relevant, which is very interesting, as participants perceive that once 
the evacuation areas are defined and evacuation is under way this information is no 
longer necessary. This is not the case as if the waves change their course, speed or 
height; the evacuation plan may need updating or changing too. This was confirmed 
later by discussions with some of the participants who mentioned it as irrelevant. 
Similarly, 31% of participants recommend ?The condition of evacuee accommodation? 
as irrelevant to ?Return phase?, whereas it is important information to decide whether 
evacuees should return or not. Finally, 39% mentioned that ?Number of returning 
evacuees? as not relevant information; this is mainly because no support is being 
provided to self-returning evacuees so keeping track of them is irrelevant.  
Table 4.2 illustrates the detailed information needs of the Operations function according 
to different phases of a mass evacuation operation.  
Table 4.2 Operations function information requirements 
Emergency 
phase 
Information requirements (Operations) % 
R 
Tsunami 
confirmed Pre 
Decision phase 
Information on current situation  100 
Update from other organizations or NCMC to keep GEOC 
updated 
100 
Information on assigned tasks by the Controller 92 
Evacuation 
decision phase 
Information on the lead organization, usually police, to lead the 
evacuation 
92 
Information on how to contact organizations like police and fire 
service 
92 
101 
 
Information on who are contact persons in various 
organizations 
100 
What is the response by external organizations 69 
Evacuation 
phase 
 
 
Require situation reports and Incident Action plans to see 
whether everything is going according to the plan 
85 
Result of previous actions/decisions 92 
The co-ordination between NCMC, GEOC, LEOC and within 
GEOC 
100 
Information on responses by external organizations to monitor 
their progress 
100 
Incident action plan is required to identify the required resources 92 
Post 
evacuation 
phase 
Progress of welfare services in evacuee camps 69 
Updated water levels to update plans 61 
Which other locations will need attention to update plans 92 
The resource usage in comparison to plan 92 
Return Phase 
The updates from organizations providing welfare  100 
Number of evacuees needing help for return 77 
Resource requirements of evacuees to return 92 
 
All the information requirements gathered for the ?Tsunami confirmed pre-decision 
phase? and ?Evacuation decision phase? are marked relevant. However, 69% of the 
participants think that the requirement ?what is the response by external organizations? 
is relevant. Moreover, the remaining 39% of participants think it is more relevant to 
?Evacuation phase? when some response from other organizations is expected. 
?Progress of welfare services in evacuee camps?, is declared relevant information to the 
?Post Evacuation phase? by 69% of the participants who suggested that it is more 
relevant for the role of Welfare function instead of Operations, whereas, after 
discussion with respondents it is affirmed as relevant since the Operations function also 
has to monitor the processes of other roles. Information requirements for all the other 
102 
 
phases are declared as relevant except ?Updated water levels?, which is redundant 
information during the ?Post evacuation phase?, since evacuation is already completed, 
however, this information is more relevant to the ?Return phase?. 
Table 4.3 shows the detailed information needs of the Logistics function during a mass 
evacuation operation. 
Table 4.3 Logistics function information requirements 
Emergency 
phase 
Information requirement (Logistics) % 
R 
Tsunami 
confirmed Pre 
Decision phase 
Information on to how to contact staff of GEOC 100 
Information on what resources are likely to be required in future 92 
Information on the process of acquiring the required resources 100 
Information on time required to arrange the required resources 100 
Evacuation 
decision phase 
Information on which resource is required in what quantity 92 
Information on available resources 100 
Information on the current location of resources 92 
Information on how to contact person or organization 100 
Information on what resources are in use and where 100 
Information on which resource requests are pending 100 
 
 
Evacuation 
phase 
 
Information on where resources are currently being used 100 
Information on resources being released to be used elsewhere 100 
Information on staff availability for planning 92 
Information on progress of evacuation so that more resources 
can be deployed if needed 
92 
Post 
evacuation 
phase 
Information on location of resources after release 92 
Information on resources deployed in the welfare camps 77 
Information on resource usage to plan future needs 92 
103 
 
Information on resources used during evacuation 100 
Return Phase 
Information on resources required for return of evacuees 92 
Information on resources beings used for return 85 
Information on resources returned to respective organizations  100 
 
The logistics function is not fully active during the ?Pre tsunami confirmation phase? as 
nothing is confirmed yet, therefore no information requirements are identified for this 
phase, although information requirements gathered for the other phases are relevant. 
This shows that the requirements gathered for the logistics function were quite 
comprehensive and appropriate.     
Table 4.4 explains the detailed information needs of the welfare function during various 
phases of a mass evacuation operation. 
Table 4.4 Welfare function information requirements 
Emergency 
phase 
Information requirement (Welfare) % 
R 
Evacuation 
decision phase 
Information on area to be evacuated 100 
Information on number of people to be evacuated 100 
Information on type of people (old, young, children) 100 
Information on  how long the resources are needed in the 
welfare camps 
100 
Information on process of resource gathering for the welfare 
camps 
92 
Contact information of people and organizations for resources 100 
Information on who needs assistance and of what type 92 
Evacuation 
phase 
Information on number of evacuees arriving at welfare camps 100 
104 
 
Post 
evacuation 
phase 
Information on resources being used in welfare camps 85 
Information on people in the welfare camps 77 
Information on future resource needs of welfare camps 100 
Return Phase 
Information on people leaving the welfare camps 93 
Information on resources used in welfare camps 85 
 
The welfare function is active only when a tsunami generation is confirmed and a 
decision for evacuation is being made at the third phase of an emergency according to 
our classification. Therefore, there are no information requirements identified for the 
first two phases. All the information requirements gathered for the remaining phases are 
marked as relevant.  
Table 4.5 illustrates the detailed information requirements of the Controller during a 
mass evacuation.   
Table 4.5 Controller function information requirements 
Emergency 
phase 
Information requirement (Controller) % 
R 
Pre-Tsunami 
Confirmation 
phase 
Information on the current situation from NCMC and other 
organizations e.g. magnitude of earthquake, location, and depth 
of source.  
100 
Overall situation picture by information from NCMC 
92 
Current response from other sectors and groups 
77 
Tsunami 
confirmed Pre 
Decision phase 
 Situation reports for approval and decision making 
85 
Incident Action Plans for approval 
92 
Evacuation 
decision phase 
Updated situation reports for approval and decision making 
85 
Updated Incident Action Plans for approval 
85 
Evacuation 
phase 
 
Information on safety of response teams and evacuees 
100 
Status of evacuated area and number of evacuees 
92 
105 
 
Post evacuation 
phase 
Evacuee status in welfare centre 
85 
Resource status in welfare centres 
77 
Return Phase 
Current situation for homes and other facilities to decide the 
return of evacuees 
77 
Prediction about future situation to decide the return of 
evacuees 
92 
 
As the Controller has to approve most of the actions and decisions most of the overall 
information goes to him/her. Except for three information requirements, all the 
information requirements are marked as relevant by 85% or more of the participants. 
Only 23% of participants think that ?response from other sectors and groups? is not 
relevant since the tsunami has been confirmed.  Similarly, ?resource status in welfare 
camps? is not relevant to the Controller as Welfare function maintain these resources 
though the Controller can approve the request if more are required.   
In survey responses, it is suggested that the ?tsunami confirmed pre-decision phase? 
does not add much to the representation of the emergency situation so  it should be 
merged with the ?tsunami confirmed evacuation phase? as once the tsunami is 
confirmed, information is gathered and processed to develop an evacuation plan that 
includes the evacuation decision. 
4.4.2 The SAVER system 
A prototype system dubbed Situation Aware Vigilant Emergency Reasoner (SAVER) is 
designed based on the SA requirements of the emergency managers during a mass 
evacuation operation. Moreover, ontology based SA is employed to meet the dynamic 
requirements of emergency situations.  
4.4.2.1 Ontology-based Situation Awareness  
Ontology is defined (Gruber, 1993) as: 
?A formal, explicit specification of a shared conceptualisation? 
Hence, ontology provides a shared vocabulary, which can be used to model a domain, 
i.e. the type of objects and/or concepts that exist in a particular domain of interest, and 
their properties and relationships with one other. As discussed in Chapter 2, Matheus et 
106 
 
al., (2003; 2005) have proposed ontology based SA systems. They also provided higher 
level ontologies to capture generic situation descriptions that can be contextualized to 
derive more domain specific ontologies. At the same time, ontology can be used to 
model an evolving situation by combining it with a reasoner that can check the validity 
of the modelled domain and draw inferences by applying rules and axioms for 
constraints, restrictions and functional dependencies (Smirnov et al., 2010). 
Figure 4.3 below shows the connection between the ontology elements and Endsley?s 
levels of situation awareness. This research extends these concepts to the use of 
ontology and inference to make projections at Endsley?s third level of SA (see green 
boxes in Figure 4.3). At the same time, it will help to integrate information from diverse 
sources (Smirnov et al., 2003).  
 
 
 
 
 
 
 
Figure 4.3 Ontology based situation awareness (Javed & Norris, 2011) 
Data received from various sources is provided to the ontology that compares it with 
the domain knowledge, infers what is relevant to the prevailing situation, and displays 
the relevant data to present Level 1 SA. Moreover, ontology relates this data with the 
domain knowledge part of the ontology and generates the possible explanation of the 
situation that is Level 1 SA. In addition to this, the reasoner infers the possible future 
states of data captured by using forward chaining inference (Blachowicz, 2009; Laudan, 
1981) on the domain knowledge and rules to generate Level 3 SA. In a forward chaining 
rule based inference system, knowledge is organised in the form of IF-Condition, 
THEN-actions. In this way, rules used by human experts can be represented in an 
expert system. When a condition is satisfied an expert makes a decision, similarly when 
Endsley?s SA 
Level 3 SA 
Level 2 SA 
Level 1 SA 
Ontology 
Inference based 
scenarios 
Relations b/w 
concepts 
Define concepts 
107 
 
all the conditions in the IF portion of a rule are satisfied, an expert system performs the 
actions in the THEN portion of that rule. Firing of a rule adds more information from 
the action part and invokes other rules. Hence, all of this chain reaction can start from 
very little initial information as an input. Moreover, this process will keep generating 
more and more information by firing various rules until the desired criteria are reached. 
The inference process goes on until either a goal is reached or no more rules can be 
fired due to the limited initial information or knowledge base or both.  
For example, consider a very simple scenario of tsunami risk after an earthquake. 
Sensors provide information on earthquake magnitude, source location, and depth. 
Moreover, DART sensors in the ocean provide wave information. The forward chain-
based inference system scans the rules sequentially and fires the first rules whose 
conditions match the contents of the working memory. Therefore, to detect whether a 
certain earthquake can generate a tsunami or not, the following general rules (see Figure 
4.4 below) and initial information are required:  
 
 
 
 
                   
 
 
 
 
Figure 4.4 Example rules for detecting tsunami generation from earthquake 
information 
Therefore, the first step in the operation of this system involves the acquisition of 
sensor readings about earthquake magnitude and earthquake source location. Once 
(a) 
Rule 1: Earthquake magnitude is high 
IF Earthquake magnitude > = 6.5 
THEN Earthquake magnitude is high 
SCAN Rule 5 
(b) 
Rule 2: Earthquake magnitude is low 
IF Earthquake magnitude < 6.5  
THEN Earthquake magnitude is low         
(c) 
Rule 3: Earthquake location is offshore  
IF Earthquake-source coordinates == 
offshore  
THEN Earthquake source location is 
offshore FIRE Rule 1  
(d) 
Rule 4: Earthquake location is onshore  
IF Earthquake- source coordinates == 
onshore  
THEN Earthquake source location is 
onshore  
108 
 
these values are available, the initial state of the working space will be as given in Figure 
4.5.  
 
 
 
 
Figure 4.5 Initial state of working space 
Now, the system will check the sensor readings and convert them to information using 
the rules 1, 2, 3 and 4. The following information (Figure 4.6) will be added to the 
working space.  
 
 
 
 
Figure 4.6 State of working space after rules scanning 
Moreover, Rule 5 given below in Figure 4.7 is fired when both Rule 1 and Rule 2 are 
fired and generates the information: ?Tsunami can be generated? 
 
 
 
 
Figure 4.7 Rule for tsunami generation  
Once it is clear that a tsunami is generated, another set of rules is fired based on wave 
parameter data from DART sensors. This data is then used to calculate the arrival time 
and probable impact of the tsunami on at risk coastal areas. Since ontology is modelled 
Working Space 
Earthquake magnitude = 7.2 
Earthquake source location = offshore 
Working Space 
Earthquake magnitude is high 
Earthquake source location is offshore 
Rule 5: Tsunami can be generated  
IF Rule 1 & Rule 2   
THEN Tsunami can be generated 
109 
 
using a formal language it is in machine-readable form and available for automatic 
processing by a computer based agent or program. The situation information modelled 
in ontology is available and it can be shared automatically with other systems. In 
addition to this, ontology provides the higher level of completeness in the description of 
procedures, required by successful automation (Viinikkala, 2004).       
4.4.2.2 Other advantages of ontology in SAVER 
There are also many other advantages of using ontology in working and the operation of 
SAVER considering the dynamic requirements of emergency situations. For example:  
1. It can solve problems of heterogeneous data input since data sources are created 
and maintained for personal and specific needs, using ontology they can be re-used 
in other tasks for other purposes by semantic integration.  
2. In large scale disasters, many organisations work together, communication and co-
ordination between them is vital for the overall combined operations. However, 
different terms have different meanings in various fields, departments and 
organisations. Therefore, it is difficult to provide all meanings in a single system. 
Ontology can solve this problem by semantically integrating different meanings to 
offer a definition that is acceptable to all. If SAVER becomes successful for a small 
group of people, by using ontology, it would be easy to extend it for multiple 
organisations interacting in emergency management.  
3. The explicit specification of domain concepts in ontology defines the meanings of 
the concepts and the constraints of their usage (Studer et al., 1998; Uschold & 
Gruninger 1996). Ontology is formally represented using a well defined language so 
that it is machine readable (Uschold & Gruninger 1996). To address the problem of 
semantic interoperability and to allow machine automation of data integration, it is 
desirable that the semantics of the data are defined explicitly and represented in a 
machine processable way. Explicit and formal ontologies can help SAVER define 
the semantics of the data, make them sharable by different users and allow machine 
processing. Hence human intervention is minimised and errors are reduced by 
decreasing their workload.  
4. Standard top level ontologies for situation modelling are available that can be used 
to develop domain specific systems. The number of these ontologies is continuously 
increasing. 
110 
 
5. Using the Web Ontology Language (OWL) to model an ontology based situation allows 
an easier and more meaningful exchange of information rather than Extensible Mark-up 
Language (XML17) alone. OWL can also be used to exchange information with some 
other standards, e.g. Emergency Data Exchange Language - Common Alert Protocol 
(EDXL-CAP). EDXL-CAP is a format for exchanging all-hazard emergency alerts and 
public warnings over all kinds of networks. CAP18 allows a consistent warning message 
to be disseminated simultaneously over many different warning systems thereby 
increasing warning effectiveness while simplifying the warning task.   
6. Ontology based systems can integrate information from diverse sources, e.g. 
ontology based schemas will not only use the table names specified in the query, but 
also the tables which are semantically the same but have different names. This is 
because OWL-aware tools can determine if two classes or individuals are the same 
or different (Tran et al., 2007). 
7. In ontology based systems, tables, in the form of classes, and relations in the form 
of properties, can be added or modified at run time if required making them very 
flexible to model evolving and dynamic situations.  
8. As more and more web applications are moving towards web 2.0, ontology based 
systems will be able to search the web effectively and efficiently for information 
needed to respond to emergency situations.  
9. With dynamic binding in ontology, a class or a sub-class can exhibit a specific 
behaviour of itself, parent class, any particular parent class, or of any other class 
depending on the type specified in a methods call at run time.  In contrast, in an 
ordinary object model concepts can only inherit the properties of parent concepts.  
10. Ontologies with limited scope can scale well in real world applications with less 
development effort since agents can populate them automatically (Shadbolt et al., 
2006). 
11. Ontology can be used to make a standard for information exchange that is 
expressive, meaningful and flexible. As standards also bring inter-operability, such a 
system can easily help in providing semantic exchange of information. 
                                                 
 
17 http://www.w3.org/XML/ 
18 http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2.html 
111 
 
Though all of the above mentioned advantages are not implemented and tested in this 
project, these can be very useful in addressing many issues related to the dynamic 
requirements of emergency situations and can easily be added if the whole architecture 
of the system is based on ontologies.  
4.4.2.3 System architecture for SAVER 
The main objective of the system design in this research is to receive information from 
various sources in real time and integrate it in a meaningful way. Other objectives 
include representation of information according to the requirements of three levels of 
situation awareness. Therefore, the system architecture is designed to receive 
information from various web sources like Common Alert Protocol (CAP) or Really 
Simple Syndication (RSS19). CAP is an XML-based data format for exchanging public 
warnings and emergencies between different alerting technologies and RSS feeds are 
used to publish frequently updated forums like news headlines, blog entries etc.  
SAVER can comprehend the meaning of the information according to the domain 
knowledge of the emergency situation. SAVER can not only model the current situation 
based on the incoming information and information requirements, it can also provide 
future states of a situation using ontology inference based on rules (as explained in the 
previous section) and constraints defined on relationships (properties) of domain 
concepts (classes). Domain knowledge is represented by using the OWL as explained in 
Chapter 3.  
The system architecture of SAVER is shown in Figure 4.8 below: 
 
 
 
 
 
                                                 
 
 
19 http://www.w3schools.com/rss/ 
112 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 4.8 SAVER system architecture for Situation assessment 
Once an earthquake occurs, its detail is posted via CAP/RSS sources20. This data is 
acquired by the SAVER and placed in a working space (computer memory) where all 
the processing is done. The data is then added to the ontology as a value property of a 
concept and lines similar to the following Figure 4.9 are inserted into the ontology file.  
 
Figure 4.9 OWL snippet showing data value assertion for earthquake magnitude 
                                                 
 
20 http://www.incident.com/cookbook/index.php/Public_CAP_Feeds 
Jena / OWL API 
Ontology 
Apply reasoner on ontology 
Apply rules / constraints  
Fact Plus 
Plus 
Semantic Web Rule Language 
(SWRL) 
CAP / RSS feeds
SA level 1:  
Display relevant data Goal 
SA level 2:  
Relate data values to model 
situation 
SA level 3:  
Predict possible future states 
of situation 
Vaadin Application 
Framework 
113 
 
This example states that an earthquake of magnitude 8.1 has occurred. Similarly, other 
details e.g. source location, depths etc., are also added. Now, a reasoner (FACT++) 
checks the changes in ontology for integrity and compares newly added values with the 
SWRL rules and OWL axioms. The rule for an earthquake incident to be classified as a 
high intensity earthquake is shown in Figure 4.10 below. 
 
Figure 4.10 OWL snippet showing rule for high intensity earthquake 
The above snippet of OWL is just a description of how an equivalent OWL class is 
defined using rules and criteria. Therefore, whenever an earthquake of magnitude more 
than 6.5 occurs it will be classified as a high intensity earthquake.  As various rules fire, 
information is added to the working space and other rules are also activated, if required. 
When the information added to the working space meets the SA requirements of a user, 
in the user ontology, information is presented to the user via a web interface.   
4.4.2.4 Ontologies developed for SAVER 
The use of ontologies is central to the design of SAVER as described in previous 
sections. The architecture uses ontologies expressed in OWL for situation modelling, 
domain modelling, situation inference and reasoning, supporting automatic information 
fusions. Please see section 4.4.2.2 ?Other advantages of ontology in SAVER? for details 
of ontology applications in SAVER. Four types of ontologies are used in SAVER:  
1. Domain Ontology 
2. Application Ontology 
114 
 
3. User Ontology 
4. Information requirements Ontology 
The reason for keeping the domain knowledge separate from the application 
knowledge and user knowledge is their potential re-usability in other contexts, as we 
will see in upcoming chapters. All ontologies are consolidated under a main SAVER 
ontology. Domain ontology uses the concepts from the field of disaster 
management specifically from the domain of tsunami risk. Therefore, it contains the 
parameters of tsunami scenario domain and relationships between the domain 
concepts. See Figure 4.11 for graphical representation for part of a domain 
ontology.     
 
Figure 4.11 Fragment of ontology showing earthquake & tsunami parameters 
Concepts like EQParameter, SourceDepth, TimeOfEarthquake, EQMagnitude, SourceLocation, 
TsunamiParameters, and AnalysedWaveHeight are from the domain ontology. An Is-a 
relationship is a way of identifying concept hierarchy. For example, EQMagnitude is-a 
kind of concept EQParameter. Figure 4.12 below shows various concepts of the User 
ontology. It describes various kinds of roles that users can have, e.g. 
Planning&IntellienceManager, ResourceManager, OperationsManagers, Controller, WelfareManager, 
etc. all of which are represented as concepts. This ontology has details of the various 
responsibilities, roles. Moreover, information requirements gathered in the previous 
section are present in information requirements ontology. SAVER uses these details by 
115 
 
performing inferences on them to find the related information requirements for a 
specific role in a specific situation.   
 
Figure 4.12 Fragment of ontology showing various roles 
Similarly, Figure 4.13 below expresses some of the concepts of the Application 
ontology. Application ontology uses the SAWA situation ontology (Matheus et al. 2003) 
on the top. The concept Situation can have different possible subtypes i.e. Situation 
BeforeCofirmation, EvacuationDecision, Evacuation, PostEvacuation, Return, PostReturn. 
Moreover, SA1, SA2 and SA3 represent three levels of situation awareness.  
 
Figure 4.13 Fragment of ontology showing situation awareness representation 
116 
 
Figure 4.14 below demonstrates the concept of time with its sub-classes in Domain 
ontology. The time class can have a different type of sub-classes i.e. 
BeforeConfirmationTime, EvacuationDecisionTime, EvacuationTime, PostEvacuationTime, 
ReturnTime etc. These different types of Time class are used to describe different phases 
of an emergency situation during a tsunami risk.   
 
Figure 4.14 Fragment of ontology showing different phases of tsunami  
Situation ontology combines all the ontologies, i.e. Domain ontology, Application 
ontology, User ontology and their extensions, which are described in upcoming 
chapters. Figure 4.15 gives an abstract view of these consolidated concepts in SAVER 
ontology.  
 
Figure 4.15 Situation ontology showing different concepts on abstract level 
117 
 
Apart from hierarchical relationships like is-a, these ontologies are also extended by 
relationships shown by +objectPropertyName between the concepts such as requires 
and hasSA. For example, the part of the ontology in Figure 4.16 shows that Role hasSA 
SituationAwareness and Role requires InformationRequirements. Since concepts SAVER and 
Role have the same object property hasSA, they will be classified as the same concept by 
the reasoner, unless they are described as disjoint concepts.  
 
Figure 4.16 Object properties showing relationships between various concepts 
Similarly, ontologies also have some more complex relations that depend on various 
conditions, e.g. Earthquake canCause Tsunami depending on some conditions i.e. 
earthquake magnitude should be high enough and its source should be offshore or close 
to the shore. These relations are also described in the form of rules and axioms as 
described in the previous section. The complete graphical representations of the 
asserted and inferred SAVER ontologies are available in Appendix F.   
118 
 
4.5 Evaluation 
SAVER is evaluated for inference capabilities using test case scenarios. The tests check 
whether inference and suggestions based on ontology based situation modelling and 
rules are correct and useful.   
4.5.1 Scenarios for evaluation  
Four different scenarios of earthquakes, different locations, different magnitudes and 
sources located at different depths are used. The reason for using different 
combinations is to see how successful ontology based modelling is in a variety of 
situations and how ontology based inference will transform incoming information into 
meaningful and useful information. The different scenarios are as shown in Figure 4.17 
below: 
 
 
 
 
 
(a)                    (b) 
 
 
 
 
 
(c)              (d)  
Figure 4.17 Four different scenarios used in simulations  
Each of these scenarios is divided into three stages according to the first three phases of 
the emergency situation. In stage 1, earthquake attributes such as magnitude, source 
location etc., are processed; in stage 2, wave information from the sensor closest to the 
Scenario 1 
Earthquake magnitude = 8.1 
Earthquake Location = 15N-115W 
Earthquake source depth = 60 km 
Scenario 2 
Earthquake magnitude = 6.8 
Earthquake Location = 28S-107W 
Earthquake source depth = 125 km            
Scenario 3 
Earthquake magnitude = 3.9                   
Earthquake Location = 4N-171W 
Earthquake source depth = 78 km           
Scenario 4 
Earthquake magnitude = 7.9                   
Earthquake Location = 21S-64W 
Earthquake source depth = 60 km 
119 
 
source location is processed. In the last stage, wave information from all the available 
active sensors is processed to see the direction of wave propagation and amplitude to 
predict the areas under tsunami threat and level of threat.  
4.5.2 Simulated output from SAVER 
The data about time of each earthquake and the consequent emergency phases are 
omitted from the tables for clarity. DART data from the sensors is also omitted here but 
details are provided in Appendix G. For all scenarios, SAVER was able to link the 
incoming data to the expected descriptions from the domain ontology according to the 
rules and constraints defined for various phases and situations. The tables below show 
the simulated output from SAVER for each scenario. Situation awareness level 1 (SA1) 
in column 3, the comprehension of situation (SA2) in column 4, and the prediction 
(SA3) in column 5, are according to the SA requirements.   
4.5.2.1 Scenario 1 
Table 4.6 SAVER?s output from scenario 1 (Javed & Norris, 2011) 
Scenario 
Number 
Emergency 
Phase 
SA1:  
Magnitude (M) 
Depth (D) 
Location (L) 
SA2 SA3 
1 Pre-
confirmation 
M = 8.1                  
D =  60 km            
L = 15N-115W  
High Magnitude 
Location is in water 
Can generate tsunami! 
1 Tsunami 
confirmation 
Pre-decision 
DART data from 
sensor near source 
Wave height is very 
high near source  
Tsunami is generated 
Potential tsunami 
threat! 
1 Evacuation 
decision 
DART data from all 
sensor until now 
Wave height is very 
high on all active 
sensors  
Impact time:13hrs  
North-east of New 
Zealand  
 
In Scenario 1, an earthquake of magnitude 8.1 is high enough to generate a tsunami and 
this magnitude data is simply classified as Level 1 SA data as it is a  relevant piece of 
information for the understanding the situation. For SA Level 2, the magnitude data is 
interpreted as high enough to generate a tsunami. Similarly, data about the co-ordinates 
of the earthquake is also regarded as SA Level 1 since it is relevant to situation 
assessment in the current scenario, and SA Level 2 assessment interprets the co-
120 
 
ordinates as an offshore or onshore location (offshore in this case). This information is 
combined using rules to generate Level 3 SA indicating that since the earthquake is high 
enough to generate a tsunami and the source is located offshore, it is likely to generate a 
tsunami. Therefore, SAVER?s suggestions on the first stage are accurate and promising.  
In the second stage of Scenario 1, data from the DART sensor close to the earthquake 
source confirms that there is a change in wave characteristics like height, amplitude etc., 
and this information qualifies as Level 1 SA since it is relevant to the understanding of 
the prevailing situation. This change in the wave information is interpreted as a tsunami 
generation and therefore tsunami generation is confirmed to provide Level 2 SA. Once, 
tsunami generation is confirmed, SAVER predicts a threat to the coastal areas as a Level 
3 SA.  
In the third stage of Scenario 1, data (SA Level 1) from DART sensors in the ocean 
shows how waves are propagating, i.e. direction and wave heights at different sensor 
locations that lead to an understanding of situations at those sensor locations providing 
Level 2 SA. This Level 2 SA, along with the knowledge of previous similar scenarios, 
becomes the basis for SA Level 3 SA to predict the expected arrival time of waves, 
height of largest waves and expected inundated areas. 
4.5.2.2 Scenario 2 
Table 4.7 SAVER?s output from Scenario 2 (Javed & Norris, 2011) 
Scenario 
Number 
Emergency 
Phase 
SA1:  
Magnitude (M) 
Depth (D) 
Location (L) 
SA2 SA3 
2 Pre-
confirmation 
M =  6.8                
D =  125 km           
L = 28S-107W  
Moderate Magnitude 
Location is in water 
Can generate tsunami! 
2 Tsunami 
confirmation 
Pre-decision 
DART data from 
sensor near source 
Wave height is very 
high near source  
Tsunami is generated 
Potential tsunami 
threat! 
2 Evacuation 
decision 
DART data from all 
sensor until now 
Wave height is very 
high on all active 
sensors  
Impact time:11hrs  
North-east of New 
Zealand  
 
121 
 
In Scenario 2, earthquake magnitude data is Level 1 SA and once compared with rules, it 
generates SA Level 2 information i.e. earthquake magnitude is high and enough to 
generate a tsunami. Similarly, the source location co-ordinates of the earthquake 
comprise Level 1 SA data, which is transformed to Level 2 SA information that these 
co-ordinates point to an offshore location. Hence, all this information is combined and 
compared with rules to generate overall SA Level 3 information that the earthquake is 
of high magnitude and its source is located offshore and therefore it is able to generate a 
tsunami. Moreover, at the next stage of scenario simulation, DART data from the 
sensor closer to the earthquake source shows that there is a change in attributes of 
waves, therefore this change is interpreted as a tsunami to provide Level 2 SA. The 
confirmation of tsunami generation predicts the threat to the nearby coastal areas 
representing Level 3 SA information.  
At the third stage of Scenario 2, data (level 1 SA) from DART sensors in the ocean 
shows a summary of the direction and height of waves at different sensor locations. The 
wave data is used to provide Level 2 SA i.e. understanding of the behaviour and 
propagation of waves at different sensor locations. To generate Level 3 SA, i.e. expected 
arrival time of waves, height of largest waves and expected inundated area, at different 
coastal areas information from Level 2 SA about direction of propagation, amplitude 
etc. is used along with the knowledge of previous similar scenarios.  
4.5.2.3 Scenario 3 
Table 4.8 SAVER?s output from Scenario 3 (Javed & Norris, 2011) 
Scenario 
Number 
Emergency 
Phase 
SA1:  
Magnitude (M) 
Depth (D) 
Location (L) 
SA2 SA3 
3 Pre-
confirmation 
M = 3.9                  
D =  78 km            
L = 4N-171W  
Low Magnitude 
Location is in water 
Cannot generate 
tsunami! 
3 Tsunami 
confirmation 
Pre-decision 
DART data from 
sensor near source 
Wave height is normal 
near source  
No tsunami threat! 
3 Evacuation 
decision 
DART data from all 
sensor until now 
Wave height is normal 
on all active sensors  
No impact, no threat!  
 
In Scenario 3, earthquake attributes, i.e. magnitude, source location and depth are 
relevant to understand the tsunami situation (Level 1 SA data). The attributes perceived 
122 
 
in level 1 SA are compared using rules to generate SA Level 2 information i.e. 
earthquake magnitude is low and insufficient to generate a tsunami. Similarly, source 
location co-ordinates of earthquake are Level 1 SA data, which is transformed to level 2 
SA information indicating that these co-ordinates point to an offshore location. All of 
this information is combined and compared with rules to generate overall SA Level 3 
information i.e. although the earthquake source is located offshore; it has too low a 
magnitude to generate a tsunami.  
Moreover, on the next stage of scenario simulation, DART data from the earthquake 
regions is still collected from the closest sensor to make sure that the tsunami is not 
generated declaring wave height is normal to present Level 2 SA. Therefore, no change 
in the DART data from the closest sensor infers that a tsunami has not been generated 
and so the Level 3 SA infers that there is no tsunami threat in the near future.  
At the third stage of Scenario 3, data (SA Level 1) from all the DART sensors in the 
ocean provides a summary of wave attributes. Since no change in the behaviour of 
waves is observed, these DART sensors will remain inactive. This absence of activity is 
seen as no tsunami generation (level 2 SA) and therefore no possible threat (level 3 SA).  
4.5.2.4 Scenario 4 
Table 4.9 SAVER?s output from Scenario 4 (Javed & Norris, 2011) 
Scenario 
Number 
Emergency 
Phase 
SA1:  
Magnitude (M) 
Depth (D) 
Location (L) 
SA2 SA3 
4 Pre-
confirmation 
M = 7.9                  
D =  60 km            
L = 21S-64W  
High Magnitude 
Location is on land 
Cannot generate 
tsunami! 
4 Tsunami 
confirmation 
Pre-decision 
DART data from 
sensor near source 
Wave height is normal 
near source  
No tsunami threat! 
4 Evacuation 
decision 
DART data from all 
sensor until now 
Wave height is normal 
on all active sensors  
No impact, no threat!  
 
In Scenario 4, incoming earthquake information is classified as Level 1 SA to give a 
summary of earthquake attributes i.e. magnitude, source location, and depth. These 
perceived attributes are compared using rules to develop Level 2 SA information i.e. 
123 
 
earthquake magnitude is high enough to generate a tsunami. In addition, the source 
location co-ordinates of earthquake (Level 1 SA data) are interpreted to develop Level 2 
SA information that these co-ordinates point to an onshore location. Therefore, once 
this information is combined and compared using the rules, Level 3 SA is generated 
showing that the earthquake is located onshore, even though its magnitude is high; it 
cannot generate a tsunami. 
On the second stage of scenario simulation, DART data from the earthquake regions is 
still collected from the closest sensor to make sure that a tsunami is not generated. In 
the current scenario, SAVER provides Level 2 SA that wave height is normal and no 
change in the DART data from the closest sensor infers that the tsunami has not been 
generated. Hence, at Level 3 SA, SAVER infers that there is no tsunami threat in the 
near future from the earthquake under consideration.  
At the third stage of Scenario 4, data from all the DART sensors in the ocean provides a 
summary of wave attributes in the form of Level 1 SA. Like Scenario 3, since no change 
in the behaviour of waves is observed, these DART sensors remain inactive. This 
absence of activity is interpreted as no tsunami generation (Level 2 SA) by SAVER and 
therefore it is inferred that there is no possible tsunami threat (Level 3 SA).  
In all the scenarios described above, SAVER has responded very well and as expected. 
It has successfully generated meaningful and useful interpretations of the incoming data. 
These interpretations are according to SA requirements and seem very promising to 
improve the decision making ability of emergency managers. More insight into the 
evaluation will be gained by the user evaluations in its actual use and are covered in the 
following chapters.    
SAVER has effectively modelled situation information on two levels - 1) domain 
knowledge to infer what type of information is required to gain various levels of SA by 
considering the SA requirements, and, 2) transforming the actual information that is 
needed to achieve all levels of SA. Therefore, it has not only fruitfully classified 
incoming information but also provided accurate inferences. Since inference uses rules, 
these rules offer a reason for every classification or suggestion so that a user can 
understand how SAVER has reached its conclusions.  
124 
 
4.6 Conclusions 
Considering the importance of SA requirements and the complexity involved in 
gathering these requirements, the cognitive task analysis tool, EGDIA, is used for 
information elicitation. Information required for developing and maintaining SA in 
relation to the goals, tasks and decision making needs of five emergency manager roles 
were elicited according to three levels of SA. The EDGIA method was quite successful 
as very few changes were recommended during validation of these requirements by a 
nationwide questionnaire. The validation process hence showed that this research has 
accurately identified the comprehensive information needs of emergency managers. In 
comparison to previous studies, this is the first known occasion that comprehensive 
information requirements of various Group Emergency Operation Centre (GEOC) 
roles are gathered for a tsunami scenario. These SA requirements became the basis for 
the design and development of a system to support a computer based SA.   
Ontology was proposed as a solution to model a dynamic situation because of its 
various applications in the field of semantic tagging of incoming information. This can 
be useful for the design of a computer based agent that can automatically perform 
information processing and fusion to give it a meaningful and useful form. Ontology 
was successfully used to design a computer based prototype dubbed SAVER that can 
model an emergency situation, apply rules and constraints to input data and their 
relationships, draw correct inferences from the assembled information and predict 
future situation states.  
The developed SAVER was evaluated to see whether the situation modelling was 
accurate and the inferences made are useful and aligned with the SA requirements.  
SAVER is tested in four simulated scenarios to see its output for the SA requirements at 
three different levels of SA. Results demonstrate that an ontology based inference can 
be used to construct systems that can support and potentially improve human situation 
awareness during emergency decision making. Moreover, all the rules were fired as 
required and accurate inferences were made as per the situation and SA requirement. 
Therefore, it was deduced that such systems can be tested in simulated experiments by 
human evaluators.  
SAVER provides situation assessment and shares the results with human decision 
makers, so they can get another view of the situation and critically review their situation 
125 
 
model, whilst saving valuable time during crisis management. Moreover, it can become 
the basis to develop an automatic information processing and fusion system. Such a 
system can help in making better decisions quickly that will reduce disruption, economic 
cost, and most importantly, loss of life.  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
126 
 
Chapter 5 
IMPROVING AN INDIVIDUAL?S SITUATION 
AWARENESS USING SAVER 
5.1 Introduction  
This chapter tackles the problem of supporting the individual?s Situation Awareness 
(SA) identified in Chapter 3. The problem is first explained in detail. In the next section, 
the suggested solutions to the problem are discussed. The chapter then explains the 
proposed design and development of the solution prototype. The design and 
development section is followed by the evaluation section. In the last section of the 
chapter, results are summarised and conclusions provided.  
5.2 Problem Identification  
In emergency management, to achieve a goal certain tasks are performed, each of which 
requires a certain level of SA for successful completion. An individual?s SA in an 
emergency situation is affected by many cognitive factors like short term memory, long 
term memory, stress level, mental models etc. (Bolstad & Endsley, 2000). These factors 
are highly dependent on the individual?s level of experience and affect SA at different 
levels (Endsley, 1995b). Getting the wrong perception of the situation elements can lead 
to the wrong decisions or mistakes in carrying out the tasks and hence endangering the 
whole operation. This is because the perception becomes the basis for the 
comprehension and projection of the situation and therefore it needs to be accurate to 
gain a correct understanding of the situation. In addition, mistakes can be made on the 
comprehension level too, i.e. while developing an understanding of the situation 
elements, and these mistakes can also lead to the wrong projection or predictions about 
the situation and can ultimately result in wrong decisions. Therefore, developing and 
maintaining accurate SA at every level is critical and important for the success of an 
operation.  Moreover, as mental models are linked with the experience and usually 
highly experienced emergency managers are scarce in emergency situations, there is a 
high risk of misunderstanding the situation by novice emergency managers. Even 
127 
 
experienced emergency managers are prone to mistakes when the stress level is high and 
time to process the information is short.  
Furthermore, stress is a conspicuous feature of emergency situations, along with 
multiple responsibilities, shortage of available time and fatigue takes over the already 
meagre short term memory where all the information processing is done. Hence, either 
it takes too long to understand the incoming information or, there is a high chance of 
missing some important information while performing a task or making a decision.  
Because of the workload and multiple roles, an operator?s comprehension of a state may 
concentrate on a specific view of information and ignore other important aspects. The 
negative impacts of some of these factors can be minimised to improve SA by 
automating repetitive tasks using a computer based systems so that these tasks are less 
attention demanding and the workload of operators is reduced. A system with static 
user requirements cannot work, as there are many different users with dynamic 
requirements. However, having multiple systems will create problems of integration and 
inter-operability. Therefore, the problem is how to design a computer based system that 
provides support to an individuals? SA, meeting their diverse and dynamic requirements.  
In emergency operations, while making sense of situations, emergency managers very 
often consult domain experts or other team members to get their point of view on the 
prevailing situation. This helps them to clarify their doubts (if any) in their own mental 
model or problem and to update their SA about the situation. Similarly, we believe that 
a computer based system, which has the perceptions of situation elements and 
semantically understands their meaning/significance, can be very helpful in updating 
human SA. This chapter describes the evaluation of such a system to understand the 
extent of improvement it can bring to the human SA. 
Since SA is affected by the way in which information is presented to the user, the HCI 
of such a system can play an important role, e.g. by attracting the user?s attention by the 
use of flashing colours, bold text or pictorial information like a graph, map or picture 
etc.  Therefore, it is important to understand the best way to present information to the 
users so that their SA can be enhanced to the maximum possible extent.   
Below is a summary of the problems discussed above in the form of questions: 
128 
 
1. How to design a computer based system supporting an SA by fulfilling diverse 
user requirements and keeping the information load to a minimum? 
2. Whether SAVER (the prototype developed in Chapter 4) can improve SA 
during a mass evacuation operation? 
3. How to design a user interface design for SAVER (the prototype developed in 
Chapter 4)? 
4. How to evaluate the user interface design developed for SAVER (the prototype 
developed in Chapter 4)? 
  
5.3 Suggested Solution to the Problem 
In Endsley?s model of SA, an individual?s SA requirements are defined as the dynamic 
information needs associated with the goals of operators (Endsley et al., 2003b; 
Wickens, 2008; Endsley, 2000). The information presented to a person through a system 
without understanding the effect of his/her goals on his/her SA has no meaning 
(Endsley et al., 2003b). Therefore, the capability of an information system to support 
emergency managers and decision makers in successfully accomplishing their tasks and 
making accurate decisions depends on how well the system supports the attainment of 
their goals (Albers, 2004). Hence, supporting their SA will mean providing them with 
the information they need to carry out their tasks to achieve their goals.     
The information requirements of different roles needed to develop and maintain their 
SA during various phases of mass evacuation operation have been gathered and 
reported in Chapter 4. These information requirements are the basis of the system 
design. To provide specific and readily useable information, SAVER should consider its 
users? contextual parameters, i.e. his/her role, responsibilities, goals and objectives, 
tasks, and more importantly, the information required to develop and maintain his/her 
SA to carry out these tasks. Hence, in other words SAVER should provide the right 
information to the right persons at the right time to achieve their goal.  This can be 
achieved by the semantic modelling of the contextual information (Chen, 2004).  
By reminding the decision makers about the information they may have overlooked, 
selectively presenting information of immediate relevance to their roles and 
requirements, and advising on the best possible course of action, can highly improve 
their decision making effectiveness and efficiency. Hence, the suggestion is to leverage 
129 
 
the effective and efficient emergency decision making by improving the SA of 
emergency managers. Moreover, the domain knowledge about the context of its users 
can be ontologically modelled to enable semantically enriched information integration 
and presentation. Therefore, the system with an understanding of the situation can work 
as a team member of emergency managers to update their SA, instead of simply a 
computer system.  
Since SAVER is intended for human use, a module of SAVER needs to be developed 
which has user interfaces for use in emergency scenario simulations. These interfaces 
should allow users to test and evaluate various SAVER features. Hence, the main 
objective of this chapter of the thesis is to provide an adequate user interface to the 
proactive information about the situation on all three SA levels. System should promote 
a more holistic view of a situation to enhance SA and to prompt its users with the 
additional information related to their current goals.  
SA oriented system design (Endsley et al., 2003), which is a User Centred Design (UCD) 
approach is considered for designing the user interface since it aims at improving the 
SA.  
5.4 Design and Development  
There are two major belief systems in Human Factors (HFs) related to the design of 
systems: the technology centred approach and the user centred approach (Endsley et al., 
2003). Traditionally, systems have been designed from a technology centred perspective. 
Alternatively, the user centred design (UCD) is a better mode of achieving more 
effective systems (Endsley et al., 2003). Following the UCD approach, an SA oriented 
system design focuses on the users? needs for the development of an information 
system that supports and improves users? SA so they can make good decisions.  
?Supporting user?s SA? is different in different domains (Endsley et al., 2003). It can be 
achieved by fulfilling the users? dynamic information needs in that particular domain 
(Endsley et al., 2003).  
Situation Awareness (SA) can answer the questions of what information is needed, how 
to maximise the performance of decision makers, what information means in a 
particular context etc. (Endsley & Rodgers, 1998; Prasanna et. al., 2009). SAVER 
provides three levels of SA and helps in the decision making process by suggesting the 
possible interpretation of the situation along with the reasons for recommending them. 
130 
 
The operation of SAVER will have a twofold benefit. If the decision maker agrees with 
the situation analysis suggested by the system, then this will increase the manager?s 
confidence in his SA and reduce delays whereas a difference between the human and 
machine recommendation will alert the managers to the aspects of a situation that 
he/she might have missed. Therefore, the objective of this chapter is to meet the 
dynamic needs of users, by providing them with highly specific and usable information.  
5.4.1 Contextual SA information  
Chen et al., (2004) defines context as: 
?Context is information about a location, its environment attributes (e.g. noise level, light intensity, 
temperature and motion) and the people, devices, objects and software agents that it contains. Context 
may also include system capabilities; services offered and sought the activities and tasks in which people 
and computing entities are engaged, and their situational roles, beliefs and intentions?.  
Therefore, according to the above definition, context information about the people, and 
their tasks, activities and situational roles can be used to distinguish them from the other 
users. Moreover, the same attributes can be used to provide the exact information 
required by these people. Hence, the systems using the context information to provide 
services are called context aware systems. Following are the relevant possible 
applications of the context aware system in the current research project.  
5.4.1.1 Possible applications of context information in SAVER design 
a.  Personalised user interface 
In the past, context information has mostly been used for user interface design in 
mobile devices (Schilit, 1995; Schilit et al., 1994; Rekimoto, 1996). More recently, many 
web based applications have also started using context information for interface design 
to meet the user needs in the best way. Some web applications, e.g. Google Inc. use 
many context parameters, the most important of these are location, tasks (email context, 
browsing or searching context) and a user?s behaviour patterns from previous usage 
(historical data).  
Similarly, in SAVER, context information is being used to provide the right information 
to the right person in the right way. Hence, the user interfaces can be optimised 
according to the user roles and tasks they are performing, for example, interface for a 
Planning and Intelligence role to develop a situation report or incident plan, interface 
131 
 
for a Controller to have an overall look on the operation or summary of areas whose 
evacuation has been completed or the remaining areas on a map etc.  
b. Adaptive behaviour   
Content adaptability: Context information can be used to generate an adaptive behaviour 
for the user depending upon various factors. Using the location, role and position 
parameters of context, emergency managers in Group Emergency Operation Centres 
(GEOC) will get different levels of information than managers in the Emergency 
Operation Centre (EOC) as GEOC looks after several EOCs. Similarly, the National 
Crisis Management Centre (NCMC) which looks after GEOCs from all over New 
Zealand do not require great detail of what is going on in the individual cities or suburbs 
but require only a summary of the whole situation.  Hence, various context parameters 
are combined to generate the different sets of users and provide an adaptive situation 
analysis for them as per their requirements. Similarly, the content can be adapted 
according to the properties and capabilities of the devices used by emergency managers.  
Content re-usability: Moreover, content developed for one purpose can be adapted and re-
used for another purpose, e.g. incoming information to update situation awareness can 
be directly added in the situation report along with the situation interpretation saving a 
lot of time for the planning and intelligence role.  
c. Intelligent applications  
Context can also be used to develop an intelligent application to facilitate the users by 
automating various processes for them. The automation reduces their workload and 
stress, both of which are the biggest challenges for emergency managers in emergency 
situations. For example, the contextual information about the job roles, their goals, and 
objectives can be used along with the SA requirements to develop information that will 
fulfil most of the requirements of SA development and maintenance.  
5.4.1.2 Acquiring context specific information  
SAVER uses context information combined with SA requirements to fulfil the 
contextualised SA requirements. Therefore, information processing can be automated 
according to the SA requirements of the emergency managers using their contextual 
information of roles, objectives, tasks and location etc., Moreover, SAVER also uses the 
contextual information for adaptive user interface and adaptive content generation as 
discussed above. Figure 5.1 below shows how various context parameters can be used 
132 
 
by the following general steps to provide the specific information to the users in a 
personalised form.   
 
 
 
 
 
 
 
Figure 5.1 Personalised information for users using contextual information 
In Figure 5.1 above, user context is acquired using inference over the following different 
clues. 
1. Identify user?s role 
2. Check the emergency phase 
3. Check user?s goals and objectives according to the user?s role and the emergency 
phase 
4. Infer the tasks and decisions that user needs to make to achieve their goals and 
objectives 
5. Gather the information requirements of the user to carry out these tasks and 
make decisions 
6. Categorize the  information according to the three levels of SA 
7. Provide the information to the user 
Acquiring the accurate context is a great challenge in context aware systems. Moreover, 
developing software tools to process the sensed contextual information is also an 
important issue (Abowd, et al., 1997; Benerecetti et al., 2001; Schilit et al., 1994). 
Therefore, this research also addresses the issues of modelling, inferring and using of 
context information to provide personalised information to emergency managers.  
1. User Role 
4. Tasks / 
Decisions 
5. Information 
requirements 
3. Goals & 
Objectives 
2. Emergency 
phase 
6. Levels of SA 
 
7. Information  
133 
 
5.4.2 Ontology based inference for contextual SA information  
5.4.2.1 Context reasoning 
Context reasoning is a method of processing the context information and making sense 
of it, i.e. making the context information useable in context aware systems. Context 
reasoning also detects and resolves inconsistent information about context. In context 
aware systems, the context of the system is automatically decided by using the context 
clues or information. To enable automatic processing of context information, it must be 
presented in machine processable form. Previously, (Asthana et al., 1994; Coen, 1998; 
Dey, 2000; Schilit et al., 1994) contextual information has been presented using data 
structures or class objects in programming languages.  
However, since these languages only provide syntax representation, they lack semantic 
representation and hence inter-operability which is considered essential for information 
sharing (Chen, 2004). Recently, Chen (2004) proposed an ontological rule based logical 
inference architecture called CoBrA to enable context reasoning. Web ontology 
language (OWL) is a perfect fit for presenting context information since it is flexible 
enough to model context information in a formal language (machine processable) and 
also allows rule based inference. Another important reason for choosing an ontology 
based approach for context reasoning is that this research is already using ontology 
based situation modelling and reasoning (Chapter 4). Therefore, both context and 
situation modelling and reasoning can have a seamless interface. One of the advantages 
of this approach is the flexibility for extension. For example, it would be very easy in 
future to add more dimensions of context information like types of devices (desktop 
computers, smart phones, Personal Digital Assistants, computer tablets etc.) used by 
users to provide personalised information according to the device features 
(Christopoulou & Kameas, 2005; Ye et al., 2007).  
5.4.2.2 Architecture design for using context information 
Below is the architecture design of SAVER using ontological contextual and situation 
reasoning.  
 
 
 
134 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 5.2 SAVER architecture design for personalised information 
In Figure 5.2 above, SAVER detects the user login and provides this information to the 
context ontology. The reasoner checks whether there is any prevailing emergency 
situation from the activities log, i.e. activities performed by emergency managers from 
the time when an earthquake occurred to the current time and infers the current state of 
the emergency situation. However, the user can also manually update the emergency 
phase status in the SAVER. The stepwise process of an automatic emergency phase 
change is as follows. Each sub-class of Time has a data property ?currentStatus? which 
can have a Boolean value, i.e. true or false. Only one phase can have ?true? as 
currentStatus and status can only be changed in sequence, i.e. from ?pre-confirmation 
Jena / OWL API 
Situation ontology 
(OWL) 
CAP / RSS field data 
Fact Plus Plus 
(Reasoner) 
SA level 1 information 
SA level 2 information 
SA level 3 information 
Context ontology 
(OWL) 
Required Information 
User role 
Emergency phase 
Goals & objectives 
Tasks & decisions 
135 
 
phase? to ?tsunami confirmed decision phase? and so on.  Table 5.1 below summarises 
various emergency phases and the status of currentStatus data property associated with it.  
Table 5.1 Boolean representation of various emergency phases 
Situation Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 
Situation normal  F F  F  F  F 
Pre-confirmation 
phase  
 T  F  F  F  F 
Tsunami confirmed 
decision phase 
F T F  F  F 
Evacuation phase F  F  T F F 
Post evacuation phase F  F  F T F 
Return phase  F  F  F F T 
 
Depending on the emergency phase information, goals and objectives are obtained from 
the context ontology and related tasks are advised to the user. SAVER not only reminds 
the user about his/her tasks and decisions but also provides the information required to 
perform the tasks and make decisions. This task/decision information is also provided 
as a supplement to the situation analysis (Chapter 4) done by SAVER. Hence, the user 
gets the pre-processed information on different levels of SA according to the 
requirement. Other parts of the architecture design have been described in the 
?architecture design? section of Chapter 4.  
5.4.2.3 Ontologies developed for contextual SA 
In this module, previously developed situation ontologies are extended to accommodate 
the contextual information. Figure 5.3 below shows an example of the partial ontology 
using contextual concepts (Goals, Tasks, Role, EmergencyPhase etc.) and their relationships 
(toAchieve, make, during, hasGoal, etc.).    
136 
 
 
Figure 5.3 Partial ontology using contextual concepts  
Figure 5.4 below shows the information requirements according to the various 
emergency phases and roles. 
 
Figure 5.4 Partial ontology showing information requirements according to the 
various emergency phases and roles 
Figure 5.4 above shows that the task e.g. prepare_incident_action_plan during the 
emergency phase decision_phase has information requirements like number_of_self_evacuees, 
137 
 
population_of_area_to_evacuate, available_resources_for_evacuation, safe_zones_to_evacuate. 
information_on_who_is_responsible_for_what etc.  
5.4.3 Human computer interaction (HCI) design  
5.4.3.1 Information presentation 
Apart from information needs of decision makers, SA requirements also focus on how 
information is integrated to tackle each decision (Endsley et al., 2003). During time 
critical and complex situations such as emergencies, the way in which information is 
presented to users influences their SA. Effective and efficient decision making depends 
on how accurately and efficiently the needed SA is developed, therefore, to speed up the 
situation assessment process, duration to acquire the needed information can be 
reduced (Endsley et al., 2003). Just presenting a mass of data will do no good and 
confuse the user who has limited time to process it and make use of it. Therefore, 
providing processed and readily usable information for a user?s SA also contributes to 
enhance their SA.  
5.4.3.2 Situation awareness oriented design principles 
SA oriented design (SAOD) approach (Endsley et al., 2003) is used to design the user 
interface. Endsley et al. (2003a) proposed 50 design principles for systems that aim to 
enhance SA and, more recently, (Bolstad et al., 2006) has prioritized eight of them as 
critical for good situation awareness. Briefly, (see the Bolstad et al. paper for details) the 
eight main principles are as follows: 
1. Organizing the information around goals: Displays and interfaces should present 
the information needed to support the decision making to achieve a particular 
goal.  
2. Present Level 2 SA information directly: Support the comprehension by 
displaying information that is processed and integrated in SA terms. Reduce the 
need for users to process information. 
3. Provide assistance for Level 3 SA: Support decision making with systems that 
are capable of projecting the future events and states of the current situation. 
4. Support global SA: Design displays that deliver the big picture (e.g. an overall 
map visible at all times) and minimise attention narrowing. 
138 
 
5. Support trade-offs between goal driven and data driven processing: A balance 
between goal driven and data driven system design is required so that both can 
complement each other.  
6. Keep the user focused on important attributes: This makes it easier for a user to 
bring their accumulated experience to bear on a situation and to amplify their 
awareness (Kaplan & Simon, 1990).  
7. Take advantage of parallel processing: Parallel systems can save time by 
processing multiple information streams and validation checks concurrently.  
8. Use information filtering carefully: Filter information to reduce overload but 
impose appropriate criteria to avoid removal of key material.  
 
These principles have been developed on the basis of theoretical models and processes 
involved in acquiring and maintaining SA in dynamic and complex environments 
(Endsley, 1990). Moreover, SA oriented design principles are applicable to a wide variety 
of system designs. They have been successfully applied as a design philosophy for 
systems involving remote maintenance operations, medical systems and flexible 
manufacturing cells (Endsley, 2001). Therefore, since current research also aims at 
improving the SA of emergency managers for decision making during mass evacuation 
operations, these principles are being used to design the user interface of SAVER.    
5.4.4 SAVER features 
A prototype user interface of the SAVER system was developed to see if the 
information provided by SAVER improved the SA of emergency managers at all SA 
levels. Moreover, it was developed to evaluate the human interaction (HCI) and 
functionality of SAVER. The SA requirements identified in section 4.4.1 indicate the 
information required to enhance various levels of SA for diverse job roles at different 
phases of an emergency. Therefore, interfaces to support all three SA levels were 
developed for evaluation. For Level 1 SA, an interface was developed to provide 
perception of information elements that were relevant and required for a particular job 
role and emergency phase. For example, Figure 5.5 shows how the interface displayed 
the Level 1 SA information for Planning and Intelligence Managers at the ?pre-
confirmation? phase of a tsunami scenario. The information about the earthquake 
magnitude, source location, depth etc. was provided to represent Level 1 SA at a pre-
confirmation phase of a tsunami. The same information summary, i.e. magnitude, 
139 
 
location and depth was also shown on the map if the cursor hovered over the icon 
indicating the earthquake.  
 
Figure 5.5 Interface showing levels of SA information about earthquake attributes 
In Figure 5.5, the red or green colour of the circle around the earthquake location 
specifies whether the earthquake magnitude is of high or low intensity respectively. This 
interpretation of situation elements represents Level 2 SA information. Furthermore, by 
showing the location on the map apart from textual information also makes clear 
whether the earthquake source is located on/offshore.  
Details of wave heights from DART data were provided in text format and the same 
wave attributes were displayed as a Level 1 SA if the cursor was placed on the sensor 
icon as shown in Figure 5.6 below. Moreover, the circle around the sensor icon provides 
Level 2 SA by indicating whether the wave height at this sensor is very high, high, 
medium, or low using red, pink, blue and green colours respectively. Moving the cursor 
on these circles also displayed a ?tool tip? note that ?wave height is HIGH? if the circle 
was pink as shown in Figure 5.6 below. For Level 3 SA, the predicted time of arrival of 
the wave along with the expected height was provided to managers so they could start 
preparations and planning for the probable evacuation while detailed inundation models 
were developed. The coloured picture on the right side provides the expected arrival 
time of the waves at different coastal areas from various tsunami sources in the Pacific 
Ocean to give an idea of the number of hours available for evacuation.   
140 
 
 
Figure 5.6 Interface showing three levels of SA information about wave information  
Moreover, Level 2 SA was provided to show the meaning of perceived information in 
the prevailing situation (see Figure 5.6 above) with colour circles around the sensor 
icons. This interpretation also explained the significance of perceived information in 
relation to goals and objectives of the job role (user). Likewise, in Figure 5.7 below, the 
blue circle with blue fill indicates the current position of propagating waves based on 
DART data.  
Similarly, Level 3 SA information was provided in the form of a prediction about the 
future states of a situation or possible evolution of the current situation. For example, in 
Figure 5.7 below, the pink area indicates the areas under tsunami risk and when a user 
puts a cursor on the border of this area, expected arrival time of the tsunami waves is 
shown.  
 
Figure 5.7 Providing comprehension and projection about the evolving situation 
141 
 
The designed system focuses on the information requirements of emergency managers 
and the flow of information during the emergencies. As time passes, new information is 
added to the existing situation model in a meaningful way to create an expressive and 
useful overall picture of a situation. Hence, this recognition of a situation by SAVER 
can be used in many different ways, e.g. 
 Updating the human SA 
 Automatically triggering other processes like planning for mass evacuation, and,  
 Informing various job roles about their tasks and responsibilities depending 
upon the prevailing situation  
Having a true understanding of the situation allows SAVER to support its human users 
by providing them with a reasoned explanation of the situation. This will help human 
decision makers to think critically about their mental models and improve their decision 
making capability. In addition, semantic understanding of an emergency situation and its 
phases permits SAVER to guide emergency managers about their responsibilities in a 
prevailing situation along with the information required for decisions. This process 
improves the efficiency of the task performance of individuals by supporting their SA 
and saves emergency managers time for other important tasks.  
5.5 Evaluation  
Supporting human users in intensively complex environments through interface design 
is a growing challenge for human factors (Kirlik and Strauss, 2006). Having many 
dimensions, SA depends on the environment and the task. Advancement in the 
technologies supporting SA depends on the advancement in SA measurement 
techniques (Salas et al., 1995). SA measurement techniques are classified into two basic 
categories: 
 Direct measurement of SA, i.e. objective real time investigations  
 Indirect measurement of SA by inferring a user?s SA based on the user?s 
functional state, behaviour, performance or subjective questionnaires.  
Our approach to assessing individual SA focusses on both direct and indirect 
approaches to evaluate SAVER by all possible means. Moreover, SAVER is also 
evaluated according to the human computer interaction (HCI) techniques. Objective 
measures of SA have been extensively validated to be reliable for various domains 
142 
 
(Fracker, 1988; Gugerty, 1997; Endsley and Rodgers, 1998). In using objective 
measures, an individual?s perception (Level 1 SA), comprehension (Level 2 SA) and 
projection (Level 3 SA) of the situation are compared with some ground truth of the 
situation. Therefore, data is collected by asking the participants about the environment, 
answers are then compared with the reality of the situation to see the number of correct 
answers. Three ways to gather objective measurement data are (Saner et al., 2009): 
 In real time, as task is being completed 
 During an interruption in the task activity  
 After the task is completed  
The best example of this approach is Situation Awareness Global Assessment 
Technique (SAGAT) (Endsley, 2000). SAGAT can be used to measure SA on all three 
levels (Endsley, 1995a). This type of measurement directly taps into the operator?s 
perception rather than inferring them from their behaviour, which may be affected by 
other factors (Endsley, 2000). Therefore, this research uses SAGAT for the direct 
measurement of SA as it is the most widely used and accepted SA measurement 
technique.  
5.5.1 Direct measurement of SA using SAGAT  
5.5.1.1 Situation Awareness Global Assessment Technique (SAGAT) 
The Situation Awareness Global Assessment Technique (SAGAT) is a tool developed 
to assess a user?s SA across all the elements of SA requirements (Endsley, 1988b, 1990). 
In a simulation, the system of interest is frozen at randomly selected times, system 
displays are blanked and users are asked about their perceptions of the situation at that 
time. SAGAT queries can be tailored to evaluate the user?s SA on all three levels of SA. 
Detailed information about a user?s SA is collected on an element basis and evaluated 
against reality, thus proving an objective assessment of the user?s SA.  
Queries across the full range of a user?s SA requirements, minimise the possible biasing 
of attention, as subjects cannot prepare for the queries in advance since they could be 
queried over almost every aspect of the situation to which they would normally attend.  
Moreover, these queries are phrased to be as similar as possible to how the person 
thinks about the information, and should not require any extra transformations or 
decisions on the part of the user (Endsley, 2000).  
143 
 
5.5.1.2 Validity of SAGAT 
The SAGAT technique has shown a high degree of validity, sensitivity and reliability for 
measuring SA (Endsley, 2000). SAGAT also has good sensitivity to the system 
manipulations, automation manipulations, expertise difference and operational concepts 
in a variety of domains (Vidulich, 2000). Moreover, findings indicate that SAGAT is 
sensitive to the changes in the task load and to the factors that affect the user?s attention 
(Endsley & Rodgers, 1998; Fracker, 1988; Gugerty, 1997). In addition, the reliability of 
SAGAT is also demonstrated by many studies (Collier & Folleso, 1995; Endsley & 
Bolstad, 1994; Fracker, 1991; Gugerty, 1997).  
5.5.1.3 SA Measurement  
In order to measure the SA using the method discussed above, 16 members of the New 
Zealand Group Emergency Operations Centre (GEOC) participated in two types of 
experiments, i.e. using SAVER and without using SAVER. The overall method of the 
experiment is the same as SAGAT (Endsley, 2000). Analyses of SA requirements (at all 
three levels of SA) were used to develop 17 questions, which were asked during four 
different freezes. A stress factor was introduced by showing the timer with every 
question. Details of queries used are provided in Appendix G. A web based application 
for questionnaires is used to gather responses since it can reduce the burden of analysis. 
Moreover, users do not have to leave the computer screen, which can result in 
forgetting the SA elements being queried. In addition, using a computerised 
questionnaire allowed the use of a virtual ticking clock along with every question to 
create the stress factor and to capture the duration of answering each question.   
Duration of response to every question is timed for comparison, as time is also an 
important factor during emergency decision making. To avoid any bias that can affect 
the results, experiments were done in two parts on two different days, which were more 
than a week apart so that the learning from one experiment did not affect the next 
session.  
5.5.1.4 Participants 
All the participants (n=16) were real emergency managers and active member of EOCs 
with two to eight years of practical experience and all of them had participated in either 
a real event or in a national exercise. Details are provided in table 5.2 below. All of them 
were familiar with the various roles of EOC and their responsibilities.  
144 
 
All the participants performed the experimental tasks. Since, SA was measured on three 
levels for diagnostic analysis; the results of each SA level were combined to see the 
overall SA at three levels. Later, overall results of each level were combined to generate 
one SA measure of each individual. The user?s score was analysed as the percentage of 
correct answers against the ground truth. Individuals were coded according to their 
experiment type, and whether he/she performed the experiment with SAVER first or 
not. For example, the code P1S1 was assigned to participant P1 if he/she used SAVER 
first whilst code P2S2 indicates that participant P2 used SAVER following a previous 
experiment.   
Table 5.2 Distribution of participants 
Participant 
(Identifier) 
Experience 
(Years) 
Participated in 
one or more 
National 
Exercises 
Participated in 
one or more 
emergency 
events 
P1 3 ? 4 YES NO 
P2 3 ? 4 YES NO 
P3 5 ? 6 YES YES 
P4 0 ? 2 YES NO 
P5 3 ? 4 YES NO 
P6 5 ? 6 YES YES 
P7 3 ? 4 YES NO 
P8 5 ? 6 YES NO 
P9 7 ? 8 YES YES 
P10 3 ? 4 YES NO 
P11 7 ? 8 YES NO 
P12 0 ? 2 YES NO 
P13 5 ? 6 YES YES 
P14 7 ? 8 YES YES 
P15 0 ? 2 YES NO 
P16 5 ? 6 YES YES 
 
145 
 
5.5.1.5 Instrument and scenario  
A tsunami scenario was used during the experiments. The individuals were supposed to 
complete a situation report about the prevailing situation. Information about the 
earthquake parameters, wave attributes like heights, location etc. were provided to 
individuals in the form of four feeds. Each of the information feeds was followed by a 
question and answer session. SAVER provided information to the users according to 
the SA oriented design principles described above and also provided some analysis of 
the incoming information as described in the section ?5.4. SAVER Features?, above. 
During the task, the session was frozen and a web based application opened a question 
in front of the participant. As soon as the question appeared on the screen, a timer also 
appeared with the question, encouraging the participant to answer as soon as possible. 
Once the participant has answered all the questions, he/she can continue with the task. 
Participants were asked directly about the situation, e.g., what is the location of the 
earthquake source? Similar questions were asked on every freeze. See Figure 5.8 below 
for an example question.  
 
Figure 5.8 Example SAGAT query 
On the other hand, when participants were not using SAVER, they were provided with 
a soft copy of a document containing information in the format provided by the United 
States Geological Survey (USGS21) website used to publish earthquake information. In 
addition, information about the wave parameters and tsunami warnings etc. was 
                                                 
 
21 http://www.usgs.gov/ 
146 
 
provided in the format of Pacific Tsunami Warning Centre (PTWC22) website that 
provides Deep Ocean Assessment and Reporting Tsunami (DART) data. See the 
example Figure 5.9 below for data provided to the non-SAVER user.  
 
Figure 5.9 Data for non-SAVER user 
To reduce learning bias, experiments were done in two parts on two different days 
(Session 1 and Session 2) more than a week apart. Out of the total sixteen participants, 
eight individuals were asked to perform the experiment with SAVER first and then 
without SAVER (Group 1), and the other eight individuals were asked to perform 
experiments without SAVER first and then with SAVER (Group 2) as shown in Figure 
5.10 below. 
                                                 
 
22 http://ptwc.weather.gov/ 
147 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 5.10 Order of the experiments  
5.5.1.6 Response variables  
Two response variables were used in the experiment:  
1. The percentage of correct responses to the SA queries targeting different levels 
of SA, i.e. perception (SA1), comprehension (SA2), and projection (SA3) 
2. The time taken to answer each question 
An SA query pool was developed to assess the possible effect of using SAVER on the 
participants? SA. Queries were designed to cover every level of SA. Level 1 SA queries 
were structured to determine if participants were able to identify and remember the 
important or critical elements of the emergency situation required to predict tsunami 
generation. Level 2 SA queries were designed to determine if participants have 
successfully analysed the situation elements as e.g. earthquake magnitude is high or low 
and wave heights are normal or high and successfully related them to their goal and task. 
Queries for Level 3 SA were designed to find if participants had successfully predicted 
the future states of the situation, e.g. possibility of situation generation, from Level 1 
Participants 
P1, P2, P5, P6, 
P9, P10, P13, 
P14 used 
SAVER 
Participants 
 P1, P2, P5, P6, 
P9, P10, P13, 
P14 used System 
B 
Participants 
P3, P4, P7, P8, 
P11, P12, P15, 
P16 used System 
B 
Participants 
   P3, P4, P7, P8, 
P11, P12, P15, 
P16 used 
SAVER 
 
Session 
1 
 
Session 
2 
Group 1 Group 2 
148 
 
and 2 SA information. The level 1, 2, and 3, queries included nine, four, and three 
questions respectively.   
Example responses to SA queries are shown in Table 5.3 below. Participant responses 
to the SA queries were graded based on the ground truth for the simulation using a five-
point scale ranging from Highly likely to Not likely at all. To get an SA measure for every 
participant at each SA level, all the query results of the participant related to that SA 
level are aggregated and averaged.  
Table 5.3 A sample of SA query pool and example responses 
SA 
Level 
Question Ground truth 
(Answers) 
1 What was the magnitude of earthquake? 8.1 
1 What was the depth of earthquake source? 28km 
1 What was the location of the earthquake source 
(coordinates)? 
15.0 W 115.0 N 
2 According to your interpretation, earthquake magnitude is 
________________ to generate a tsunami. 
High Enough 
2 Was Earthquake source offshore or onshore?  Offshore  
2 According to your understanding of source depth and 
location, what is the likelihood that it can cause a tsunami? 
Likely 
3 According to your understanding of situation, how likely is 
tsunami generation? 
Likely  
 
5.5.2 Experiment results 
5.5.2.1 Validity of the experiment method 
To ensure that participants in the second session were not influenced by learning from 
the first, the performance of the two groups in Session 1 are compared with their 
performance in session 2 as illustrated in Figure 5.11. The results of these two groups 
were cross checked using independent sample T-test using significance level p<0.05. No 
significant difference between their performances was found (t = -0.56, p-value = 0.577) 
confirming that the performance of the participants (M = 86.6, S.D = 8.90) was not 
affected by the previous experiment session (M = 84.8, S.D = 9.10), where M stands for 
Mean and S.D stands for Standard Deviation.  
149 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 5.11 Order of experiments red circles showing the two groups of participants  
However, the possibility still exists that the average percentage across Group 1 and 
Group 2 in Session 1 is the same as the average percentage across these groups in 
Session 2. Therefore, further tests were conducted to confirm that participants did not 
use the learning from the previous session as it is very critical for the valid evaluation of 
SAVER. The performance of SAVER users who performed the experiment first 
(Group 1, Session 1) is compared with the performance of those SAVER users who 
performed the experiment in the second session (Group 2, Session 2) as shown by the 
orange circles in Figure 5.12 below.  The results of this comparison using independent 
sample T-test revealed there is no significant difference (t = -1.10, p-value = 0.28) 
between the performance of both the groups (M = 90.5, S.D = 5.97 for using SAVER 
in the first session; and M = 93.0, S.D = 3.65 for using SAVER in the second session).   
Moreover, the performance of non-SAVER users who performed the experiment first is 
compared with the performance of those non-SAVER users who performed the 
experiment in the second session as shown by the green circles in Figure 5.12 below.  
The results of this comparison using independent sample T-test also proved that there is 
Participants 
P1, P2, P5, P6, 
P9, P10, P13, 
P14 used 
SAVER 
Participants 
 P1, P2, P5, P6, 
P9, P10, P13, 
P14 used System 
B 
Participants 
P3, P4, P7, P8, 
P11, P12, P15, 
P16 used System 
B 
Participants 
   P3, P4, P7, P8, 
P11, P12, P15, 
P16 used 
SAVER 
 
Session 
1 
 
Session 
2 
Group 1 Group 2 
150 
 
no significant difference (t = -0.21, p-value = 0.83) between the performance of both 
groups (M = 79.2, S.D = 8.31 for the non-SAVER users in the first session; and M = 
80.0, S.D = 7.58 for the non-SAVER users in the second session).   
 
 
 
 
 
 
 
 
 
 
Figure 5.12 Comparing performance of the SAVER users (orange circles) and non-
SAVER users (green circles) in the first session with the second session 
5.5.2.2 SA measurement by score with and without SAVER 
The results of the individual SA measurements (see Figure 5.13 below) shows that the 
individuals who used SAVER performed very well compared with those who did not 
use it. Moreover, paired sample T-test was conducted and a significant difference (t = -
2.84, p-value = 0.008) was found between the score without using SAVER (M=73.0, 
S.D=10.5) and the score of participants using SAVER (M=82.58, S.D=8.40).   
Participants 
P1, P2, P5, P6, 
P9, P10, P13, 
P14 used 
SAVER 
Participants 
 P1, P2, P5, P6, 
P9, P10, P13, 
P14 used System 
B 
Participants 
P3, P4, P7, P8, 
P11, P12, P15, 
P16 used System 
B 
Participants 
   P3, P4, P7, P8, 
P11, P12, P15, 
P16 used 
SAVER 
 
Session 
1 
 
Session 
2 
Group 1 Group 2 
151 
 
            
 
                                                                                                                                      
Figure 5.13 Comparison of Individual SA measures with and without SAVER 
Results are further broken down into the three levels of SA. Therefore, answers for the 
questions targeted on each level of SA are isolated according to the SA level and the 
percentage averaged to provide the user?s SA on that SA level. Figure 5.14 below 
presents Level 1 SA for the individuals. 
 
Figure 5.14 Comparison of the SA Level 1 measures with and without SAVER 
The result of measuring a shared Level 1 SA using and without using SAVER, shows 
that there is an improved performance by those who were using SAVER during the 
0
20
40
60
80
100
120
P1
S1
P2
S1
P5
S1
P6
S1
P9
S1
P1
0S
1
P1
3S
1
P1
4S
1
P7
S2
P8
S2
P3
S2
P4
S2
P1
1S
2
P1
2S
2
P1
5S
2
P1
6S
2
SA
 M
ea
su
re
 %
 
Participants 
SA Measure 
SA using SAVER
SA Without SAVER
0
20
40
60
80
100
120
SA
 L
E
ve
l  
1 
M
ea
su
re
 %
 
Participants 
SA Level 1 Measure 
SA1 with SAVER
SA1 wihtout SAVER
152 
 
experiment. Paired sample T-test shows that a significant difference (t = -3.26,  p-value 
= 0.003) is found in the score of participants using SAVER (M=88.4, S.D=8.94) and 
the score for individuals not using SAVER (M=78.5, S.D=6.84). Figure 5.15 below 
presents Level 2 SA for the individuals.   
 
Figure 5.15 Comparison of the SA Level 2 measures with and without SAVER 
Figure 5.15 shows the results of measuring Level 2 SA with and without SAVER. The 
results show that use of SAVER improves SA Level 2. Paired sample T-test shows that 
a significant difference (t = -3.90, p-value = 0.001) is found in the score of participants 
not using SAVER (M=82.8, S.D=17.6) and the score for using SAVER (M=93.7, 
S.D=7.39). Figure 5.16 below presents Level 3 SA for the individuals.   
 
Figure 5.16 Comparison of SA Level 3 measures with and without SAVER 
0
20
40
60
80
100
120
P1
S1
P2
S1
P5
S1
P6
S1
P9
S1
P1
0S
1
P1
3S
1
P1
4S
1
P7
S2
P8
S2
P3
S2
P4
S2
P1
1S
2
P1
2S
2
P1
5S
2
P1
6S
2
Sa
 L
ev
el
 2
 M
ea
su
re
 %
 
Participants 
SA Level 2 Measure 
SA2 using SAVER
SA2 without SAVER
0
20
40
60
80
100
120
P1
S1
P2
S1
P5
S1
P6
S1
P9
S1
P1
0S
1
P1
3S
1
P1
4S
1
P7
S2
P8
S2
P3
S2
P4
S2
P1
1S
2
P1
2S
2
P1
5S
2
P1
6S
2
SA
 L
ev
el
 3
 M
ea
su
re
 %
 
Participants 
SA Level 3 Measure 
SA3 using SAVER
SA3 without SAVER
153 
 
Figure 5.16 shows the results of measuring SA Level 3 with and without SAVER. The 
result again shows that SAVER improves SA at Level 3. Paired sample T-test shows 
that a significant difference (t = -3.90, p-value = 0.001) is found in the score of 
participants not using SAVER (M=78.7 S.D=14.44) and the score of participants using 
SAVER (M=93.7   S.D=7.39).  
The results above show that the individual SA on all three levels was improved by using 
SAVER during the experiments. It appears that the SAVER?s SA-oriented design 
improved SA by providing the individuals with information in a format that helped 
them to improve their attention and also remember the details by supporting their short 
term memory with relevant information processing. Similarly, by sharing the 
interpretation of a situation and the prediction about the evolving situation, updated the 
individual?s SA. Moreover, the SA of SAVER users was also better than the non-
SAVER users because the demand on an individual?s cognitive resources was lower 
when SAVER was doing most of the processing for them, and when they were 
considering its recommendations and suggestions for answering the questions.  
This last observation was also supported by using other questions in the survey in which 
users were asked to rank the sources that they used every time they answered complex 
questions. Most of the users who were not using SAVER, ranked previous experience as 
their first priority and communication as the second, whereas most of the individuals 
who used SAVER preferred SAVER?s suggestions to their own experience. However, 
this result could be attributed to the fact that the experiment was a simulation and not a 
real event.  
5.5.2.3 Analysis of SA measure by time  
Since time is an important factor in emergency decision-making, it is also scarce in such 
situations. By reducing the time required to develop the needed level of SA for decision 
making or carrying out a task, the overall emergency management operations can highly 
improve. This can enable effective and efficient decision making and task completion in 
a short time. The time taken by participants to answer every question was recorded to 
see whether there is some difference between the response time of SAVER and non-
SAVER users in understanding the questions about a situation. A graph in Figure 5.17 
below shows the comparison of average time taken by every participant to answer each 
154 
 
question, with and without using SAVER. Moreover, it also shows the count of wrong 
answers.   
  
Figure 5.17 Comparison of average time taken by a user to answer the queries 
and count of wrong answers using and without using SAVER 
In Figure 5.17 we can see that the time to answer each question has considerably 
reduced when users were using SAVER, almost all the users were able to recall the 
situation elements, their comprehension and projection more quickly using SAVER 
than without using SAVER. Therefore, time to answer queries was relatively less since 
SAVER?s suggestions were helping them to understand the situation quickly. The last 
point was also confirmed by the answer to one of the HCI evaluation questions, in 
which users were asked whether SAVER helped them in quickly recalling the situation 
parameters and almost all of the users agreed. A paired sample t-test shows that a 
significant difference (t = -12.06, p-value = 0.000) is found in the average time taken by 
the users to answer queries, using SAVER (M=28.3, S.D=1.85) and without using 
SAVER (M=35.6, S.D=1.90).  
The graph in Figure 5.17 also shows the count of numbers of mistakes made by each 
user. It can be seen the users made a relatively small number of mistakes when they 
were using SAVER than while not using SAVER.  
0
5
10
15
20
25
30
35
40
45
P1
S1
P2
S1
P5
S1
P6
S1
P9
S1
P1
0S
1
P1
3S
1
P1
4S
1
P7
S2
P8
S2
P3
S2
P4
S2
P1
1S
2
P1
2S
2
P1
5S
2
P1
6S
2
N
ub
er
 o
f m
is
ta
ke
s 
&
 A
ve
ra
ge
 ti
m
e 
(S
ec
s)
 
Participants 
Time comparson for SA measure  
Avg time using SAVER (secs)
Total wrong using SAVER
Avg time without using
SAVER (secs)
Total wrong without using
SAVER
155 
 
5.5.2.4 Analysis of SA measure by user experience 
Experience is an important factor in emergency decision making since SA required by a 
decision maker can be highly influenced by his/her experience level (Endsley, 1995a; 
Klein, 1999).  However, as discussed in the ?Problem identification? section of this 
chapter, experienced emergency managers are scarce in emergency situations and they 
can also make mistakes in highly dynamic and complex situations. Therefore, SAVER is 
designed to directly update a user?s SA on all three levels and it is intended to equally 
support experienced and novice emergency managers. Data about the experience of 
participants (emergency managers) was collected before the experiments using the 
ranges of 0 to 2 years, 3 to 4 years, 5 to 6 years, 7 to 8 years, or more than 8 years of 
experience. Responses show that all the participants have experience between 0 to 8 
years so that experienced and novice emergency managers were divided into two main 
groups i.e. 0 to 4 years and 5 to 8 years. This grouping is shown in Table 5.4 below: 
Table 5.4 Grouping of participants based on years of experience  
Participant 
(Identifier) 
Experience 
(Years) 
Participated in 
one or more 
National 
Exercises 
Participated in 
one or more 
emergency 
events 
Experience 
(Years) 
P2 3 ? 4 YES NO 
0 - 4 
P4 0 ? 2 YES NO 
P15 0 ? 2 YES NO 
P1 3 ? 4 YES NO 
P10 3 ? 4 YES NO 
P12 0 ? 2 YES NO 
P7 3 ? 4 YES NO 
P5 3 ? 4 YES NO 
P13 5 ? 6 YES YES 
5 ? 8 
P8 5 ? 6 YES NO 
P11 5 ? 8 YES NO 
P9 7 ? 8 YES YES 
P16 5 ? 6 YES YES 
P6 7 ? 6 YES YES 
P3 5 ? 6 YES YES 
P14 7 ? 8 YES YES 
 
156 
 
Moreover, all the emergency managers had experience of actively participating in one or 
more national level exercises. Almost 80% of the emergency managers who have 4 to 8 
years of experience have also participated in one or more real emergency events. 
Whereas, none of the emergency managers with experience of 0 to 4 years had 
participated in a real emergency event. The graph in Figure 5.18 below shows the 
performance of both groups using SAVER in overall SA measure.   
 
Figure 5.18 Comparison of SA performance based on experience using SAVER 
Figure 5.18 shows that the performance of emergency managers with more experience 
(5 to 8 years) is slightly better than those with lesser experience (0 to 4 years). However, 
the comparison using independent sample T-test shows that there is no significant 
difference (t = -1.61, p-value = 0.13) in the performance of both groups of emergency 
managers. Hence, SAVER has supported the novice emergency managers (M=89.9, 
S.D=4.99) to a degree that their performance is brought near to the performance of 
experienced emergency managers (M=93.7, S.D=4.46).  
To confirm it further, the performance of the emergency managers without using 
SAVER is tested based on the experience level. Figure 5.19 shows the performance of 
emergency managers without using SAVER based on experience.         
 
60
70
80
90
100
110
1 2 3 4 5 6 7 8
SA
 M
ea
su
re
 %
 
Users  
SA Measure % Based on Experience - Using SAVER 
Experience (0-4)
Experience (4-8)
157 
 
 
Figure. 5.19 Comparison of SA performance based on experience without 
SAVER 
The performance of emergency managers with more experience (5 to 8 years) is 
significantly better than those who have lesser experience (0 to 4 years). Moreover, 
independent sample T-test shows that there is a significant difference (t = -2.71, p-value 
= 0.01) in the performance of both groups of the emergency managers (M=75.2, 
S.D=7.55 for novice and M=83.9, S.D=5.12 for experienced emergency managers). 
This confirms the argument that the SA of decision makers is affected by their 
experience levels (Endsley, 1995a; Klein, 1999). Therefore, the effect of using SAVER 
becomes clear in improving the performance of both the novice and experienced 
emergency managers. 
5.5.3 Human Computer Interface (HCI) evaluation of SAVER 
5.5.3.1 HCI Experiment design 
Human Computer Interface (HCI) of a system plays an important role in improving the 
SA of the users (Endsley & Jones, 2001). Evaluation of HCI is therefore important to 
make sure the system is usable and useful. The emergency managers evaluated the HCI 
of SAVER by filling in a questionnaire after the experiments, based on usability and 
functionality (Bevan, 2001; ISO 9126-1, 1998). 
The following measureable attributes of usability as described in (ISO 9126-1, 1998; 
Folmer & Bosch, 2004) were used:   
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8
SA
 M
ea
su
re
 %
 
Users 
SA Measure % Based on Experience - Without SAVER 
Experience (0-4)
Experience (4-8)
158 
 
 Ease of use is the capability of software that enables the user to believe that 
using the software requires less effort to use (Bevan, 2001; Davis, 1989).  
 Understandability is the capability of the software product that enables the user 
to understand the role of software in the given conditions.  
 Learnability is the capability of the software product that enables the user to 
learn its use to perform the desired task  
 Operability is the capability of the software product that enables the user to 
operate and control it. 
The following measureable attributes of functionality as described in (ISO 9126-1, 1998; 
Folmer & Bosch, 2004) were used:  
 Accuracy is the capability of the software that provides the correct and desired 
information as expected by the user for the given conditions.  
 Suitability is the capability of the software that enables the user to recognise 
whether the use of software is suitable for the given task or not. Moreover, it 
describes the tasks and conditions of the software use. 
 Usefulness is the capability of the software that enables the user to perceive 
whether the use of the software improved his/her performance (Davis, 1989) 
i.e. user?s effectiveness and efficiency to perform his/her task.   
Though usefulness and overall functionality of SAVER have already been evaluated in 
the previous section and Chapter 4, this section evaluates the perceived usefulness, 
accuracy, and suitability of SAVER by asking users to rate the attributes on a scale. 
Moreover, the HCI of SAVER is also evaluated in terms of subjective queries about the 
SA of the user. These queries were targeted at all three levels of SA and users were 
asked about the degree of their SA at different levels of SA.  Figure 5.20 below 
summarises the measureable attributes of HCI factors used in the evaluation.  
 
 
 
 
159 
 
 
 
 
 
 
 
 
 
 
 
Figure. 5.20 HCI factors and their measurable attributes  
A questionnaire comprising 18 questions attempted to address all the measurable 
attributes described above. The order of the questions was deliberately mixed to avoid 
random ratings by the participants. However, in the analysis, the questions are described 
according to the HCI factors (usability, functionality, SA subjective measure). Each 
question was rated on a seven-point Likert scale where 1 was ?strongly disagree? and 7 
was strongly agree.  Figure 5.21 below shows the example queries about the HCI of 
SAVER.  
 
Figure. 5.21 The example queries about the HCI of SAVER 
Usability  
Ease of use 
Understandability 
Learnability  
SAVER HCI Functionality 
Accuracy  
Suitability  
Usefulness 
Subjective SA  
SA level 1 
SA level 2 
SA level 3 
160 
 
5.5.3.2 Usability questionnaire responses  
The overall usability of SAVER is measured by asking the questions about each attribute 
of usability as described above. Table 5.5 below shows all the questions relating to the 
usability of SAVER. Question U1 was intended to measure the ?learnability? attribute 
of usability, whereas questions U2 and U3 were anticipated to measure the ?ease of use? 
attribute of usability metric. In order to check the degree of the understandability 
attribute of usability, questions U4, U5, and U6 were used.  
The responses to question U1 show that the participants found it very easy to learn the 
use of SAVER. A one sample T-test against the neutral rating value 4.00 confirmed 
these interpretations (t = -5.96, p-value < 0.001). In addition, responses to the questions 
U2 and U3 reveal that SAVER was very easy to use and navigation through the system 
was very straight forward (t = -5.83, p-value < 0.001 for question U2; t = 4.25, p-value 
= 0.001 for question U3). Moreover, responses to the questions U4, U5, and U6 
indicate that the suggestions provided by SAVER about the perception, comprehension 
and projection of the situation are very easy to understand (t = 10.95, p-value < 0.001 
for question U4; t = 12.65, p-value < 0.001 for question U5 and t = 11.22, p-value < 
0.001 for question U6).  
Table 5.5 Responses to questions related to the usability of SAVER  
Question 
Number 
HCI Attribute Question / Statement  Response 
Mean 
U1 Learnability Lot of training was needed to use 
SAVER 
2.6   
(disagree) 
U2 Ease of use SAVER was easy to use 5.2        
(agree) 
U3 Ease of use Navigation was easy to follow 5.1        
(agree) 
U4 Understandability  Suggestions provided were easy to 
understand 
6.0        
(agree) 
U5 Understandability Comprehension of the situation 
provided was easy to understand  
6.0        
(agree) 
U6 Understandability Projection of the situation provided 
was easy to understand  
5.43      
(agree) 
 
161 
 
All these results show that the participants have positively rated all of the attributes of 
usability. This result confirms that SAVER is easy to use and learn and its suggestions 
are all understandable.  
5.5.3.3 Functionality questionnaire responses  
In order to know how the emergency managers perceived the functionality of SAVER, a 
questionnaire addressed all the fundamental attributes of the functionality like accuracy, 
suitability and usefulness. The questions F1, F2, and F3 were intended to measure the 
suitability aspect, questions F4 and F5 were anticipated to measure accuracy aspects, and 
questions F6, F7, F8, F9, and F10 presented aspects of usefulness. All these questions 
are described in Table 5.6 below.  
Table 5.6 Responses to questions related to the functionality of SAVER  
Question 
Number 
HCI Attribute Question / Statement Response 
Mean 
F1 Suitability  SAVER guided me to improve my 
Situation awareness 
5.6         
(agree) 
F2 Suitability Given suggestions were relevant 
(according to the task) 
5.6        
(agree) 
F3 Suitability Given suggestions were timely 
(according to situation) 
5.4        
(agree) 
F4 Accuracy  SAVER interface flow was as expected  5.4        
(agree) 
F5 Accuracy  SAVER suggestions were as expected 5.5        
(agree) 
F6 Usefulness  SAVER helped me in quickly recalling 
situation elements 
4.7        
(agree) 
F7 Usefulness SAVER helped me quickly 
comprehending the situation 
5.0        
(agree) 
F8 Usefulness SAVER helped me quickly projecting 
(predicting) about situation 
4.7        
(agree) 
F9 Usefulness SAVER helped me achieving my tasks 
effectively  
5.0         
(agree) 
F10 Usefulness SAVER helped me achieving my tasks 
efficiently  
5.0        
(agree) 
 
 
162 
 
Generally, the responses to all the questions about functionality of SAVER were 
positive. One sample T-test against the neutral rating value 4.00 for each question is 
used to establish the interpretation of the responses. The responses to question F1 show 
that the participants perceived that SAVER has significantly improved their SA (t = 
7.00, p-value < 0.001). In addition, responses to questions F2 and F3 reveal that 
SAVER?s suggestions were very relevant according to the task (t = 6.48, p-value < 0.001 
for question F2) and situation (t = 7.06, p-value < 0.001 for question F3) indicating the 
overall suitability of SAVER for the given task in the provided situation.  
Moreover, responses to questions F4 and F5 illustrate the flow of operation and the 
outputs of SAVER were accurate (t = 11.22, p-value < 0.001 for question F4 and t = 
11.61, p-value < 0.001 for question F5). In addition, the responses to the questions F6, 
F7, F8, F9 and F10 suggest that SAVER has considerably helped them in quickly 
recalling the situation elements (t = 3.5, p-value = 0.003 for question F6), 
comprehending the situation (t = 3.78, p-value = 0.002 for question F7), and predicting 
about the situation (t= 2.81, p-value = 0.01 for question F8).  Furthermore, the 
responses to questions F9 and F10 demonstrate that SAVER has highly supported the 
participants in performing their tasks effectively (t = 5.5, p-value < 0.001 for question 
F9) and efficiently (t = 4.14, p-value = 0.001 for question F10). All these results confirm 
that SAVER is functioning properly in the aspects of accuracy, suitability, and 
usefulness and according to the expectations of its users (emergency managers).      
5.5.3.4 SA subjective measure questionnaire responses  
The questions listed in Table 5.7 were intended to ascertain what participants perceived 
about the degree of their own SA on each level of SA after using SAVER. A subjective 
measure of SA is usually calculated by self-rating techniques like Situation Awareness 
Rating Technique (SART) (Taylor, 1990). SART is performed post trials and involves 
the participant rating various aspects (familiarity of situation, focusing of attention, 
quantity of information etc.) of SA on a seven point rating scale (1 low and 7 high) in 
order to gain a subjective measure of SA (Salmon et al., 2009).  Question ?SA1? was 
anticipated to measure the perceived degree of Level 1 SA, question ?SA2? was 
intended to measure the perceived degree of Level 2 SA and question ?SA3? was 
presented to measure the perceived degree of Level 3 SA of the participants.  
163 
 
The overall responses to the questions show that the degree of perceived SA of each 
participant was very high after using SAVER. The responses to question SA1 show the 
participants? supposition that they had a high degree of awareness of the situation 
elements (t = 5.83, p-value < 0.001 for question SA1). The replies to question SA2 
reveal the participants? assumption that they have comprehended the situation elements 
very well (t = 10.43, p-value < 0.001 for question SA2). Furthermore, answers to 
question SA3 indicate the participants? belief that they predicted the situation very well 
during the experiment using SAVER (t = 6.26, p-value < 0.001 for question SA3). All 
these results illustrate that the degree of perceived SA at every level was very high.  
Table 5.7 Responses to related subjective measure of SA using SAVER  
Question 
Number 
Subjective SA 
measure 
Question / Statement  Response 
Mean 
SA1 SA level l I correctly identified the situation 
elements 
5.2        
(agree) 
SA2 SA level 2 I correctly comprehended the situation 
elements  
5.9           
(agree) 
SA3 SA level 3 I correctly predicted the future states 
of situation 
5.1         
(agree) 
 
5.6 Summary and Conclusions 
In emergency management, to achieve a goal, certain tasks are performed, each of which 
requires a certain level of SA for successful completion. An individual?s SA in 
emergency situations is affected by many cognitive factors like short and long term 
memory, stress level, mental models etc. These factors are highly dependent on the 
individual?s level of experience and affect SA at different levels. Getting the wrong 
perception of the situation elements can lead to wrong decisions. Moreover, as the 
mental models are linked with experience and usually the highly experienced emergency 
managers are scarce in emergencies; there is a high risk of misunderstanding the 
situation by novice emergency managers. Use of a computer based system can address 
these issues but such solutions come with problems such as how to design a system 
providing support to an individual?s SA, whether a computer based SA can improve an 
individual?s SA and how to design and evaluate the interface of such systems.  
164 
 
The capability of an information system to support decision makers depends on how 
well the system supports their goals. The contextual parameters, i.e. his/her role, 
responsibilities, objectives and tasks are also very important to provide the right 
information to the right people at the right time to achieve their goals.  
Instead of a computer based system which provides information on request, a computer 
system is needed which can work as a team member and share information proactively 
with the understanding of the information and logical reasoning. Hence, automatic 
processing of contextual information is needed. A contextual information module and 
interface is designed and developed for SAVER (prototype developed in Chapter 4) to 
be used and evaluated by the emergency managers. SAVER uses ontological contextual 
modelling and reasoning. Ontologies provide the detailed semantics needed for 
automatic contextual information processing and reasoning. To design user interfaces 
for improved SA, SA oriented system design, which is a User Centred Design (UCD) 
approach, is used.  
Since SAVER is intended to improve an emergency manager?s SA, it is necessary to 
evaluate its effect by experiments. Measuring SA techniques are classified into direct or 
objective measurement of SA and indirect or subjective measurement of SA. The SA 
measurement technique used for direct measurement of SA is Situation Awareness 
Global Assessment Technique (SAGAT). SAGAT is used to measure SA on all three 
levels. This type of measurement directly taps into the operator?s perception rather than 
inferring them from behaviours. Hence, SAGAT is the most widely used and accepted 
SA measurement technique 
The results of evaluation show that the individual?s SA on all three levels was improved 
by using SAVER during the experiments. It seems that SAVER?s SA-oriented design 
improved SA by providing individuals with the information they needed to understand 
the situation. Moreover, by processing the information on their behalf also supported 
their cognitive resources like short term memory. Similarly, sharing the interpretation of 
the situation and prediction about the evolving situation, updated the individual?s SA. 
Though this can be confirmed by further experiments for various roles. 
Time is a very critical factor in emergency decision making. Reducing the time required 
to develop the needed level of SA for decision making or carrying out a task can highly 
improve overall emergency management operations. Analysis of the time taken by the 
165 
 
individuals to answer question about SA shows that time to answer queries was 
significantly less when the participants were using SAVER than when they were not. 
This can be because of SAVER?s suggestions, which helped them to understand the 
situation quickly.  
To dig deep into the effect of SAVER on an individual?s SA, data analysis is also done 
on the basis of the user?s experience, since the experience is an important factor in 
acquiring and maintaining SA. Participants are divided into two groups, i.e. novice and 
experienced emergency managers having experience of 0 to 4 years and 5 to 8 years 
respectively. Results show that the performance of the emergency managers with more 
experience (5 to 8 years) is slightly better than those with less experience (0 to 4 years). 
However, the difference in the performance of the two groups is not significant. Hence, 
the performance of the novice emergency managers is significantly improved to the 
level of experienced emergency managers. In addition, the performance of the 
experienced emergency managers using SAVER was considerably better than the non-
SAVER users. However, when SA performance of the non-SAVER users is analysed it 
is noted that there is a considerable difference between the novice and experienced 
emergency managers. This was also confirmed by the independent sample T-test since 
the SA of decision makers is highly affected by their experience levels. Therefore, the 
improvement in the SA of novice and experienced emergency managers using SAVER 
is evident.    
The user interface of SAVER is evaluated in terms of its usability and functionality. The 
measurable attributes of the usability used in this evaluation are; ease of use, 
understandability, learnability, and operability. Whereas the measureable attributes of 
functionality used in the evaluation of SAVER are accuracy, suitability, and usefulness. 
Results of the user interface evaluation show that the participants have positively rated 
all the attributes of usability confirming that SAVER is easy to use and easy to learn and 
its suggestions are all understandable. Furthermore, results from functionality evaluation 
also approve that SAVER is functioning properly in the aspects of accuracy, suitability, 
and usefulness.  
SA is also measured by the indirect or subjective method namely Situation Awareness 
Rating Technique (SART). Responses to the SA queries show that all the SAVER users 
166 
 
perceived a high degree of SA on all three levels i.e. perception, comprehension, and 
projection.  
Hence, all the research objectives set for this chapter are achieved by answering all the 
questions mentioned in the ?problem identification? section of this chapter. SAVER?s 
contextual suggestions about the situation and information needed to develop and 
maintain SA have been proved useful by the detailed evaluation of SAVER. Moreover, 
the user interface of SAVER has also been evaluated positively. This proved that the 
design was successful in improving SA using a computer based system. Such a system 
with detail context information of all the involved roles and their responsibilities can 
considerably improve the decision making capabilities of emergency managers and can 
highly support them in performing their tasks effectively and efficiently.  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
167 
 
 
IMPROVING SHARED AND TEAM SITUATION 
AWARENESS USING SAVER 
6.1 Introduction  
Multiple teams, e.g. Police, Healthcare, Emergency Managers usually manage large scale 
emergencies such as a tsunami or volcanic risk in highly populated cities etc. Consensus 
on important decisions and large collaborative tasks require effective co-ordination and 
co-operation in situations that are complex, dynamic, unfolding rapidly and invariably 
stressful. Close co-ordination within and between teams is essential since the failure of a 
single link can risk the whole operation. Computerised decision support systems can 
facilitate and improve co-ordination and decision making by presenting, structuring, 
processing, and interpreting huge amounts of information in a short span of time.  
However, the power of such systems is enhanced even further if they are designed to 
improve the situation awareness (SA) of individual managers, their shared situation 
awareness (SSA) and team situation awareness (TSA). The goal is to ensure that team 
members have a comprehensive understanding of the situation not just for their 
individual roles but also of the roles of their colleagues 
This chapter describes the design, development, and evaluation of the SAVER 23 
extension to support SSA and TSA. The first section discusses the requirements of SSA 
and TSA and suggests a method to acquire them. The next section then explains the 
design and development of the solution prototype in the form of an extension to 
SAVER. The subsequent section evaluates SAVER to see if it supports the SSA and 
TSA of emergency managers. Conclusions are described in the last section.  
                                                 
 
23 Prototype system developed in Chapter 4 and Chapter 5 
168 
 
6.2 Problem Identification  
Emergency managers working in teams may have one overall goal or objective but they 
can have different sub-goals defined by associated responsibilities and tasks that feed 
into the main goal. Figure 6.1 shows an example in which five emergency managers 
(EM 1 to 5) perform sub-tasks 1-4 that contribute to sub-goals 1-4. In the figure, 
manager EM6 performs sub-task 5, which feeds into sub-task 1 shared with EM1.  
 
 
 
 
 
 
 
 
 
Figure 6.1 Overall goal and task division for a team of emergency managers 
For the tasks which need multiple input from team members, e.g. sub-task 4,, team 
members should know which of their colleagues are doing what, and what they need 
and when, so they can provide the right information at the right time to the right 
person. However, as dozens of organizations can be involved in emergency operations it 
is not practical for an individual to remember all persons, teams, organizations and their 
related tasks along with relevant details. Requesting this information from other team 
members can involve considerable delays (MCDEM, 2008b; MCDEM, 2011).  
Such collaborative tasks require emergency managers to share a common understanding 
of a situation so they can co-ordinate their decision making and actions. Different 
assessments of the situation can lead to unco-ordinated or even counter-predictive 
behaviour. The degree to which team members possess the same understanding of a 
Overall goal 
Sub-goal 1 Sub-goal 2 Sub-goal 3 Sub-goal 4 
Overall Task 
Sub-task 1 Sub-task 2 Sub-task 3 Sub-task 4 
Sub-task 5 
Emergency Managers (EM) 
EM 6 EM 3 EM 5 EM 2 EM 4 EM 1 
169 
 
situation is defined as shared situation awareness (SSA) as it refers to the overlap 
between members? individual SA. 
However, this overlap in itself may not be sufficient to achieve the required response to 
an emergency. Each team member?s sub-goals demand the SA needed to discharge their 
role (responsibilities) in relation to other?s activities. The degree to which every team 
member possesses this SA is known as team situation awareness (TSA). The TSA thus 
depends upon high levels of both individual SA and shared SA amongst members 
(Endsley & Jones, 1997, 2001). With low SA, individual members have a poor 
understanding of the situation, and with low SSA they tend to focus only on their own 
roles ignoring their team contribution and the actions of their colleagues. This not only 
results in the repetition of various activities but it can also delay or curtail tasks leading 
potentially to the failure of the whole mission due to the inability of team members to 
integrate their individual and shared awareness. 
From the above, TSA can be viewed as an aggregate of every member?s SA. Another 
form of TSA, which is very important for collaborative tasks, is the belief about the 
other team members? SA. For example, consider a team of two members. A and B, 
working together on a task. If member A believes that member B has misunderstood 
the situation, then the mistake can lead to failure of the task. If A?s belief is correct 
about B?s SA then A can update B?s SA by providing him/her with the required 
information or by convincing him/ her about the situation.  
When an event is catastrophic and mass evacuation is required, teams must decide if and 
when to evacuate and which areas to vacate etc. The team consensus about the 
prevailing situation and its implications needed for such decisions may be difficult to 
achieve due to the members? diverse roles and their cultural, background, and 
experience differences that make them understand and assess the situation differently. 
This problem becomes more complex as the number and type of roles involved in the 
team increases. 
Considering all the problems discussed above, a need rises for a computer based system 
that considers the varied requirements of team members and provides them with an 
easily understandable analysis of the situation; that is a system that can support the 
shared SA and team SA of managers in emergency situations. However, the design and 
170 
 
development of such computer based systems need answers to some very fundamental 
questions. These questions are summarised below: 
a. How are SSA and TSA developed for team decision making and collaborative 
tasks? 
b. How to support SSA and TSA using a computer based system? 
c. Can a computer based system improve the SSA and TSA of emergency 
managers? 
6.3 Suggested Solution to the Problem 
SSA and TSA can be improved by a computer based system that provides information 
to support the development and maintenance of the ?degree of awareness?. The first 
step in designing such a computer system is therefore to identify the information 
requirements of SSA and TSA.  
According to Endsley and Jones (2001), awareness of the team member?s requirements is 
viewed as SSA. Therefore, by reminding the team members about the other members? 
requirements will improve the SSA of the whole team. Since SAVER can store the detailed 
information requirements of all team members, its design can be extended to support SSA 
by reminding the members about the requirements of their colleagues in a timely manner. 
This can be done by reasoning over the common goals, tasks and information requirements 
to reveal and install the necessary information flows. In this way, the workload of 
emergency managers will significantly reduce since they will not be required to remember 
the details of other members, organizations or roles that are dependent on them.  
To support the common understanding of the situation, which is called TSA by Salas et 
al. (1995), SAVER can provide explicit situation assessment along with the reasoning. 
The reasoning in the form of logical arguments can clarify any doubts in the minds of 
emergency managers and they can be convinced about the prevailing situation. 
Moreover, providing them with the correct explanation of the situation generates a 
common and shared understanding among the team. Sharing of the member?s SA will 
improve the other team members? confidence (if there is a match) or otherwise it will 
give them a chance to critically analyse the situation. A second look at the situation will 
enable team members to correct their own or another team member?s SA. Sharing of 
the team member?s SA will also improve the belief about the fellow team member?s SA, 
which is regarded as TSA by Shu and Furuta (2005).  
171 
 
Therefore, SSA and TSA can be supported and improved by a slight extension to the 
architecture of SAVER. The existing architecture can provide personalised information 
to the emergency managers if it has knowledge of the prevailing situation and its users? 
context (Chapter 5). The extension of semantic modelling of the situation and the user 
context can then promote automatic and proactive sharing of the information 
requirements between users who have given their consent for distribution. Automatic 
sharing of information without a team member?s intervention reduces workload and 
task duration and improves the effectiveness and efficiency whilst proactive reminders 
can encourage members to supply information that SAVER cannot provide 
automatically. The information required to develop and maintain SSA can also be re-
used from the information requirements gathered earlier (Chapter 4).  
6.4 Design and Development  
This section focuses on the design and development of the extension for SAVER to 
support SSA and TSA. Since, an individual?s SA can be supported by fulfilling his/her 
dynamic information needs in a particular domain (Endsley et al., 2003), the same can 
be done for SSA and TSA. The following section discovers the information needed to 
develop the SSA and TSA during the mass evacuation operation following a tsunami.  
6.4.1 SSA and TSA requirements elicitation 
Information requirements for various roles of emergency managers have been identified 
in Chapter 4 using a cognitive task analysis technique, Extended Goal Directed 
Information Analysis. Although these information requirements were gathered for 
supporting an individual?s SA, some additional information was also collected like the 
possible sources of information and where the processed information will go etc. 
Therefore, by keeping the definitions of SSA and TSA in view, information flows 
between the team members can be analysed to identify the SSA and TSA requirements.  
6.4.1.1 SSA and TSA in Group Emergency Operation Centres (GEOCs)  
As discussed earlier in Chapter 2, this thesis considers Co-ordinated Incident 
Management System (CIMS) as a team architecture since it is used by emergency 
organisations in New Zealand for managing the response to an incident involving 
multiple agencies. The five basic functions of CIMS are Controller, Planning & 
Intelligence (P & I), Operations, Logistics and Welfare. In all emergency operation 
centres, including the National Crisis Management Centre (NCMC), Group Emergency 
172 
 
Operation Centres (GEOCs) or Local Emergency Operation Centres (LEOCs), these 
functions exist either in the form of sub-teams or individuals depending upon their scale 
of responsibilities. All centres co-ordinate situation and collaborative tasks according to 
their roles and responsibilities. Therefore, SSA and TSA are required at different levels. 
Figure 6.2 shows the generic information flow and co-ordination needed to carry out 
the collaborative tasks and decision making during an emergency event.  
 
 
 
 
 
 
 
 
 
 
  
Figure 6.2 Coordination between NCMC, GEOC, and LEOC  
Within NCMC, GEOC, and LEOC, the structure of CIMS defines the responsibilities 
associated with each role. These responsibilities and tasks provide the basis for 
standardising the communication channels and information flows. Moreover, this 
classification of responsibilities and their respective requirements provide the 
framework for understanding where SSA and TSA are needed in the emergency 
operations. Figure 6.3 shows the CIMS structure within NCMC, GEOCs and LEOCs 
and an overview of the respective responsibilities of various roles within the functions. 
In Figure 6.3, SSA and TSA in the GEOC are required in almost every process where 
there is more than one individual. For instance, within the P & I function, the situation 
National Crises Management Centre (NCMC) 
Local Emergency Operation Centres (LEOCs) 
Group Emergency Operation Centres (GEOCs) 
GEOC 1 GEOC 2 GEOC 3 ??? GEOC n 
Decisions & 
Advice Situation 
Reports & 
Requests 
Decisions & 
Advice Situation 
Reports & 
Requests 
LEOC 1 LEOC 2 LEOC 3 LEOC n ???
173 
 
reporting section gathers data from various sources and discusses the technical and 
scientific analysis with qualified scientists in the Scientific Advisory Group (SAG). As a 
result, their SA is updated and they develop and share a common understanding (TSA) 
of the situation before reporting it in their situation report (SitRep). 
In the case of a tsunami scenario, at the first ?pre-confirmation of tsunami? phase (see 
Chapter 4), SAG guides the Situation Reporting function about the possibility of tsunami 
generation after analysing the earthquake parameters and other historic data. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 6.3 Example of Shared and Team Situation Awareness in GEOC 
Controller Function 
Group 
Controller 
Controller 
Assistant 
Operations Function 
Operations 
Manager 
GEOC ? 
Coordination 
Communication 
Section 
Documentation 
Section Welfare Function 
Welfare Manager 
Welfare Support 
Planning & Intelligence 
P & I Manager 
Scientific Advisory 
Group 
Situation 
Reporting Section 
Advanced Action 
Plan Section 
Logistic Function 
Logistic 
Manager 
Supply Section 
Finance Section Resource 
Tracking Section 
SSA & TSA 
174 
 
The common or shared understanding of the situation is then passed to the Controller 
function so that more people share the understanding. Once the threat is confirmed, the 
Controller approves the full scale activation of the GEOC. The Controller also approves 
the SitRep or recommends changes before sharing it with other functions inside the 
GEOC or outside, e.g. with other GEOCs or NCMC. These other functions of GEOC, 
e.g. Operations, Logistics, and Welfare then take appropriate planning steps depending 
upon their responsibilities and their understanding of the SitRep.  
Once a tsunami is confirmed, areas under risk are identified along with the arrival time 
of initial tsunami waves with the help of SAG and computer based simulations. The 
whole team of functions is taken on board to decide the evacuation plan depending on 
the available resources, i.e. human resources, transport, welfare camps etc. SSA and TSA 
are developed in terms of understanding the situation and the division of 
responsibilities. Depending upon the outcome of the evacuation decision, an Incident 
Action Plan (IAP) is developed by the Planning section of the P & I function. The IAP 
is discussed with the Situation Reporting section and the P & I manager to develop the 
needed level of SSA and TSA. Once approved by the Controller and NCMC, the IAP is 
shared within the GEOCs and LEOCs for action. During the implementation of the 
IAP, SSA is required to co-ordinate the activities within GEOCs and outside, e.g. with 
NCMC, concerned LEOCs, Police, Fire Service, NGOs and other Lifeline 
organizations. By extending the required co-ordination and collaboration, these 
organisations can be considered as a team of teams having certain aspects of SSA and 
TSA needed to achieve the common goal of successful mass evacuation as shown in 
Figure 6.4. The organisations mentioned here are just examples since dozens of 
organisations can be involved in a mass evacuation depending upon its scale.  
6.4.1.2 SSA and TSA information requirements 
Based on SA oriented design principles (Endsley et al., 2003) the following 
considerations are relevant to the design of SSA and TSA oriented system:  
1. Avoid information overload in shared displays 
2. Provide flexibility to support shared SA across tasks and team members 
3. Support communication of comprehensions and projections across teams 
4. Standardisation of information flows and formats  
175 
 
5. Support transmission of SA within positions by making status of elements and 
states explicit 
6. Proactive sharing of information to save time 
7. Sharing SA of team members as and if required by other team members instead 
of broadcasting  
These design principles, and the design aspects previously described in Chapters 4 and 5 
are used to extend SAVER?s functionality to support SSA and TSA. In addition, 
information required by the team members is also shared according to their contexts 
and requirements. Tables 6.1-6.3 show some example GEOC roles to illustrate the SSA 
requirements at different levels. 
Table 6.1 SSA Level 1 requirement of Situation Reporting Section (P&I) 
SSA Requirement Template 
SSA requirement identifier SSAL1G1T1I1 
SA Level SSA Level 1 
Goal Identifier G1 
Goal Report current situation to the stakeholders 
Task Identifier T1 
Task Develop SitRep 
Information requirement identifier I1 
Information requirement Magnitude, Source depth, Source coordinates, Time 
Source (Priority) US Geological Survey (USGS) report on earthquake (1) 
Destination Situation Reporting (P & I) 
 
The Level 1 SSA information requirement in Table 6.1 can be automatically fulfilled by 
SAVER, since it can be acquired from the USGS website and reported to the managers 
performing situation reporting. Similarly, Table 6.2. provides an example of the Level 3 
SSA information requirement. 
 
 
 
17
6 
                              
F
ig
ur
e 
6.
4 
T
ea
m
 o
f t
ea
m
s 
in
vo
lv
ed
 in
 m
as
s 
ev
ac
ua
ti
on
E
xt
er
na
l o
rg
an
iz
at
io
ns
 
N
C
M
C
 
G
E
O
C
 1
 
G
E
O
C
 2
 
?
?
.?
?
?
G
E
O
C
 n
 
L
E
O
C
 1
 
L
E
O
C
 2
 
L
E
O
C
 n
 
?
?
.?
?
?
?
C
on
tr
ol
le
r 
P&
I 
W
el
fa
re
 
O
pe
ra
tio
ns
 
L
og
is
tic
s 
Po
lic
e 
 
 
??
 
Fi
re
-S
er
vi
ce
 
 
 
??
 
Sa
lv
at
io
n 
A
rm
y 
H
ea
lth
ca
re
 
 
 
??
 
177 
 
Table 6.2 SSA Level 3 requirement of Situation Reporting Section (P & I) 
SSA Requirement Template 
SSA requirement identifier SSAL3G1T1I4 
SA Level SSA Level 3 
Goal Identifier G1 
Goal Report current situation to the stakeholders 
Task Identifier T1 
Task Develop SitRep 
Information requirement identifier I4 
Information requirement Areas marked for evacuation  
Source (Priority) Scientific Advisory Group (SAG) (1), USGS (2)  
Destination Situation Reporting (P & I) 
 
Templates provided in Tables 6.1 and 6.2 are used to gather the SSA requirements for 
each level. SAVER has an information requirements broker designed to prompt team 
members about the information needs of their colleagues. When information 
requirements cannot be automatically fulfilled by SAVER, an alert is generated for 
manual input. 
However, instead of dictating the process, SAVER guides the situation assessment and the 
final decision is left for human decision makers. Users can agree or disagree with SAVER?s 
suggestions and the system shares the degree of agreement or disagreement with relevant 
people so that team members can see other?s views along with the suggestions. In this way, 
team members? beliefs about other?s understanding are also updated to improve TSA (Shu 
and Furuta, 2005). These features require information about the team members who share 
common goals, tasks and decisions. This is crucial for sharing the explicit situation 
assessment by SAVER and the SA of the fellow team members. Table 6.3 provides an 
example template of the information requirement of TSA.  
Table 6.3 TSA Level 2 Requirement of Situation Reporting (P & I) 
TSA Requirement Template 
TSA requirement identifier TSAL2G1T4I8 
SA Level TSA Level 2 
Goal Identifier G1 
Goal Report current situation to the stakeholders 
178 
 
Task Identifier T4 
Task SitRep Development  
Information requirement identifier I8 
Information requirement  Tsunami generation confirmation  
SA Source role Scientific Advisory Group (SAG) 
SA Destination role(s) Situation Reporting, P & I Manager, Controller  
 
Table 6.3 shows a situation assessed by the SAG (SA source) that needs to be shared 
with Situation Reporting, the P & I Manager and the Controller to develop a consensus 
on the prevailing situation. Hence, the SA of SAG members is shared with the 
destination roles to develop a common understanding and improve belief about team 
members? understanding.  
6.4.2 Ontology based SSA and TSA oriented system design 
6.4.2.1 Ontology-based information requirements and SA broker 
 The information requirements broker works using ontologies. Since information, 
requirements of individual roles are already present in the form of ontologies (Chapter 
4), it is easy to re-use them for SSA and TSA. Ontology-based context modelling 
(Chapter 5) makes the ontology-based information broker an ideal candidate to route 
information requirements to the team member who can fulfil them whilst offering 
seamless interaction with the rest of the SAVER architecture. Information about the 
source of each requirement is added according to its priority so that if the required 
information is not available from one source, SAVER can look for the information 
elsewhere. In this way, SAVER either supplies the required information from its 
knowledge base or prompts for manual input. SAVER also has a chat client that can be 
used to contact other team members.  
SAVER uses the following steps to perform information brokerage and improve SSA: 
1. Identify a user?s information requirements using their context information 
(discussed in Chapter 5) 
2. Identify the common information requirements (information requirements 
linked to more than one role in ontologies)  
3. Identify the sender (source) and receiver of each information requirement 
179 
 
4. Send the required information or request sender to send the required 
information 
5. Route the response from sender to receiver 
SAVER uses the following steps to perform information brokerage for improving TSA: 
1. Share the explicit situation assessment with users according to their 
requirements (improves common understanding of the situation)  
2. Share a team member?s understanding of the situation with other team members 
(improves common understanding and belief about other team members? SA) 
3. Provide the information required for every team member?s SA needed to fulfil 
his/her responsibilities (improves individual SA and hence the overall TSA)  
Figure 6.5 shows the system architecture of this module of SAVER to improve the SSA 
and TSA of the emergency managers.  
 
 
 
 
 
 
 
 
 
Figure 6.5 SAVER architecture to support SSA and TSA 
As shown in Figure 6.5, SAVER?s architecture is extended to accommodate the support 
for SSA and TSA by adding the feature to share information between the team 
members. The information requirements broker uses an ontology-based reasoner 
(Fact++) to find which information to share with whom on the basis of user context 
features, e.g. role, common goals, tasks and information requirements. Similarly, an SA 
Sharing SA 
Share team 
member?s SA 
Request team member?s 
SA requirement 
Receive team member?s 
SA requirement 
Provide SA 
requirement 
SA Broker 
Information 
Requirements 
Ontology 
User Context 
Ontology 
Domain 
Ontology 
Information 
Requirements 
Broker 
Situation 
Assessment 
 
180 
 
broker uses an ontology reasoner for identifying whose SA should be shared with 
whom, depending upon their roles, SA requirements, common tasks and decisions. 
Existing ontologies of information requirements and user?s context are re-used as 
shown in the partial example ontology in Figure 6.6. Planning&Intelligence, and Operations 
are the roles (context information) and NCMC_warnings, LEOC_sitrep and 
area_to_evacuate are the information requirements. ScientificAdvisoryGroup (role) is the 
sender or InformationRequirementSource of area_to_evacuate and Operations is 
InformationRequirementSource of NCMC_warnings and LEOC_sitrep. Information 
requirement, e.g. area_to_evacuate is shared using the other context information like 
DecisionPhase. NCMC_warning, LEOC_sitrep, area_to_evacuate are individuals of class 
(Concept) type InformationRequirment whereas hasSource, hasInformationRequirments and 
during are object properties showing relationships between the concepts. 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 6.6 Partial example ontology for TSA and SSA support  
NCMC_warning 
InformationRequirement 
hasSource 
LEOC_sitrep 
area_to_evacuate 
hasInformationRequirement 
Planning&Intelligence 
Role Operations 
ScientificAdvisoryGroup 
InformationRequirementSource 
hasSubclass 
hasSubclass hasSubclass 
hasSubclass 
hasSubclass 
hasSubclass 
during DecisionPhase 
Class Object Property Individual hasSubclass 
181 
 
6.4.2.2 SAVER User interface features for SSA and TSA support  
Explicit situation assessment for common awareness: To support the common 
awareness required for TSA, SAVER provides the explicit situation analysis with logical 
arguments (situation assessment provided in Chapter 5), and providing the reasoning 
with the situation assessment updates the user?s SA. In other words, it promotes a true 
and common understanding of the situation amongst the decision makers. Figure 6.7 
shows a SAVER screen shot that demonstrates explicit situation assessment. 
 
Figure 6.7 Explicit situation assessment provided by SAVER 
Sharing a team member?s SA for common awareness: Moreover, if the user has a 
conflict over the situation assessment, they can make their argument in the chat window 
to let others know their opinion. Otherwise, by simply selecting ?Agree? and pressing 
the ?Submit? button, their SA is shared with other relevant team members. Figure 6.8 
shows an example message that relevant team members might receive. This message will 
appear on their screen for a few seconds and then disappears but it can be viewed at any 
time in the shared SA report box as shown in Figure 6.8.  
182 
 
 
Figure 6.8 Example message from SAVER about team member?s SA 
Sharing a team member?s SA requirements to support SSA: SAVER provides the 
information about the SA requirements of the team members where manual input is 
required from the other team members. In this case, it asks for the desired information 
proactively on behalf of the person who needs it, for example, see Figure 6.9. 
 
Figure 6.9 Example message from SAVER asking for information 
Similarly, SAVER shares the information about the SA requirements automatically, if 
possible. For example, Figure 6.10 shows an alert from LEOC 1 that a new SitRep has 
arrived with update information.  
183 
 
 
Figure 6.10 Example message from SAVER about new information 
Chat client to support SSA and TSA: If further clarification is needed from another 
team member on any matter, e.g. information requirement or situation assessment, the 
chat client can be used. It allows messages to specific team members or a general 
broadcast. Moreover, it can be used to share additional information on situation 
assessment or a disagreement. Figure 6.11 shows an example message in a chat box.  
 
Figure 6.11 Example chat client for communication 
184 
 
6.5 Evaluation 
Since this module of SAVER is intended to improve the SSA and TSA of a team, it is 
evaluated by the direct measurement of SSA and TSA of SAVER users. The shared and 
team SA measurement methodology is based on SAGAT (as discussed in Chapter 5) by 
modifying it according to the requirement of Shared SA and team SA measurement.  
SSA and TSA queries were presented to the participants in the same experiments 
performed to measure individual SA (Chapter 5). An SSA and TSA queries pool was 
developed by using Endsley?s definitions of SSA (Endsley & Jones, 2001), the mutual 
belief model of TSA (Nonose et al., 2010; Shu & Furuta, 2005), and the definition of 
TSA due to Salas et al., (1995). 
Our evaluation method builds upon the shared SA outcomes identified by Jones and 
Endsley (2002), from which we propose a similarity measure that calculates the degree 
of SA shared between two team members. TSA is evaluated by directly comparing any 
two given team members with regard to the similarity of their understanding of the 
situation elements that are relevant to both of them. Similarly, SSA is measured by 
comparing their answers about the common SA requirements.  
6.5.1 Experiment design  
6.5.1.1 Instrument and scenario 
A few additional queries about the SSA and TSA were asked in the same experiments 
carried out to measure individual SA as discussed in Chapter 5. For detailed evaluation, 
SSA and TSA were measured at all three levels so the queries were designed for these 
levels as discussed in the previous section. Participants were queried about their own and 
team member?s SA requirements. Example SSA queries are shown in Table 6.4.  
In Table 6.4, the participants are asked about their own SA requirements on all three 
levels. The participants are then asked about their team members? information 
requirements. For the experiments without SAVER, ?SAVER? was not considered as a 
valid answer to a query and is not shown in the answer options in Table 6.4.  
One of the frameworks to measure team SA (TSA) is based on the team cognition and 
mutual belief model (Nonose et al., 2010; Shu & Furuta, 2005, Kano, 2007). The model 
is composed of three layers of mental components, which represent the structure of 
mutual belief in team cognition. The first layer represents an individual?s cognition, the 
185 
 
second layer represents beliefs in a partner?s cognition, and the third layer represents a 
belief in a partner?s belief (Nonose et al., 2010). Assuming there is a team of two 
members A and B, the model is described as: 
Ma = A?s cognition, Mb = B?s cognition 
Ma? = A?s belief about Mb, Mb? = B?s belief about Ma 
Ma??  = A?s belief about Mb?, Mb??  = B?s belief about Ma? 
Where in the last line the double index terms can be written: 
Where Ma??  means A's belief about what B believes about A 
And Mb??  means B's belief about what A believes about B 
Team SA evaluation is based on the degree of agreement on these beliefs and 
cognitions. For example, matching A?s cognition of the situation with B?s belief about 
A?s cognition can give a complete picture of team SA. Though the model proposed by 
Nonose et al. (2010) seem promising, they used a subjective approach to gather the data 
by asking subjects to rate their own SA. However, the subject?s belief about his/her own 
SA can be different from his/her actual SA. 
18
6 
  
T
ab
le
: 6
.4
 E
xa
m
pl
e 
SS
A
 q
ue
ri
es
 a
t o
w
n 
an
d 
te
am
 m
em
be
r?
s 
re
qu
ir
em
en
ts
 
SS
A
 
L
ev
el
 
T
ar
ge
t 
Q
ue
st
io
ns
 
P
os
si
bl
e 
an
sw
er
s 
SS
A
 
L
ev
el
 
T
ar
ge
t 
Q
ue
st
io
ns
 
P
os
si
bl
e 
an
sw
er
s 
1 
O
w
n 
SA
 1
 
re
qu
ire
m
en
t 
W
ha
t i
nf
or
m
at
io
n 
el
em
en
ts
 
do
 y
ou
 th
in
k 
ar
e 
re
le
va
nt
 in
 
th
e 
cu
rr
en
t s
itu
at
io
n?
 
E
ar
th
qu
ak
e 
m
ag
ni
tu
de
, 
So
ur
ce
 lo
ca
tio
n,
 S
ou
rc
e 
D
ep
th
 
1 
T
ea
m
 m
em
be
r?s
 
SA
 1
 r
eq
ui
re
m
en
t 
W
ha
t i
nf
or
m
at
io
n 
el
em
en
ts
 d
o 
yo
u 
th
in
k 
ar
e 
re
le
va
nt
 to
 y
ou
r 
te
am
 m
em
be
r?s
 r
ol
e?
 
E
ar
th
qu
ak
e 
m
ag
ni
tu
de
, 
So
ur
ce
 lo
ca
tio
n,
 S
ou
rc
e 
D
ep
th
  
1 
O
w
n 
SA
 1
 
re
qu
ire
m
en
t 
W
ha
t i
nf
or
m
at
io
n 
cu
es
 d
id
 
yo
u 
pe
rc
ei
ve
d 
w
er
e 
re
le
va
nt
 
to
 th
e 
ts
un
am
i g
en
er
at
io
n?
 
E
ar
th
qu
ak
e 
m
ag
ni
tu
de
, 
So
ur
ce
 lo
ca
tio
n,
 S
ou
rc
e 
D
ep
th
 
1 
T
ea
m
 m
em
be
r?s
 
SA
 1
 r
eq
ui
re
m
en
t 
W
ha
t i
nf
or
m
at
io
n 
cu
es
 d
id
 y
ou
r 
te
am
 m
em
be
r 
pe
rc
ei
ve
 a
s 
re
le
va
nt
 to
 ts
un
am
i g
en
er
at
io
n?
 
E
ar
th
qu
ak
e 
m
ag
ni
tu
de
, 
So
ur
ce
 lo
ca
tio
n,
 S
ou
rc
e 
D
ep
th
  
2 
O
w
n 
SA
 2
 
re
qu
ire
m
en
t 
W
ha
t d
id
 y
ou
 r
eq
ui
re
 to
 
un
de
rs
ta
nd
 /
 in
te
rp
re
t t
he
 
pe
rc
ei
ve
d 
cu
es
? 
Pl
ea
se
 li
st
 
so
ur
ce
s 
K
no
w
le
dg
e 
of
 ts
un
am
i 
ge
ne
ra
tio
n,
 E
xp
er
ie
nc
e,
 
co
m
m
un
ic
at
io
n,
 o
th
er
 
2 
T
ea
m
 m
em
be
r?s
 
SA
 2
 r
eq
ui
re
m
en
t 
W
ha
t d
id
 y
ou
r 
te
am
 m
em
be
r 
re
qu
ire
 to
 u
nd
er
st
an
d 
/ 
in
te
rp
re
t t
he
 p
er
ce
iv
ed
 c
ue
s?
 
Pl
ea
se
 li
st
 s
ou
rc
es
 
K
no
w
le
dg
e 
of
 ts
un
am
i 
ge
ne
ra
tio
n,
 E
xp
er
ie
nc
e,
 
co
m
m
un
ic
at
io
n,
 o
th
er
 
2 
O
w
n 
SA
 2
 
re
qu
ire
m
en
t 
W
ha
t h
av
e 
yo
u 
us
ed
 to
 
un
de
rs
ta
nd
 o
r 
in
te
rp
re
t t
he
 
pe
rc
ei
ve
d 
in
fo
rm
at
io
n?
 R
an
k 
by
 p
ut
tin
g 
a 
nu
m
be
r 
E
xp
er
ie
nc
e,
 S
A
V
E
R
, 
co
m
m
un
ic
at
io
n,
 o
th
er
 
2 
T
ea
m
 m
em
be
r?s
 
SA
 2
 r
eq
ui
re
m
en
t 
W
ha
t h
as
 y
ou
r 
te
am
 m
em
be
r 
us
ed
 to
 u
nd
er
st
an
d 
or
 in
te
rp
re
t 
th
e 
pe
rc
ei
ve
d 
in
fo
rm
at
io
n?
 
R
an
k 
by
 p
ut
tin
g 
a 
nu
m
be
r 
E
xp
er
ie
nc
e,
 S
A
V
E
R
, 
co
m
m
un
ic
at
io
n,
 o
th
er
 
3 
O
w
n 
SA
 3
 
re
qu
ire
m
en
t 
W
ha
t i
nf
or
m
at
io
n 
di
d 
yo
u 
ne
ed
 to
 p
re
di
ct
 th
e 
ts
un
am
i 
ge
ne
ra
tio
n?
 P
le
as
e 
lis
t 
so
ur
ce
s 
K
no
w
le
dg
e 
of
 ts
un
am
i 
ge
ne
ra
tio
n,
 E
xp
er
ie
nc
e,
 
co
m
m
un
ic
at
io
n,
 o
th
er
 
3 
T
ea
m
 m
em
be
r?s
 
SA
 3
 r
eq
ui
re
m
en
t 
W
ha
t i
nf
or
m
at
io
n 
di
d 
yo
ur
 
te
am
 m
em
be
r 
ne
ed
 to
 p
re
di
ct
 
th
e 
ts
un
am
i g
en
er
at
io
n?
 P
le
as
e 
lis
t s
ou
rc
es
 
K
no
w
le
dg
e 
of
 ts
un
am
i 
ge
ne
ra
tio
n,
 E
xp
er
ie
nc
e,
 
co
m
m
un
ic
at
io
n,
 o
th
er
 
3 
O
w
n 
SA
 3
 
re
qu
ire
m
en
t 
W
ha
t h
av
e 
yo
u 
us
ed
 to
 
pr
ed
ic
t t
he
 p
er
ce
iv
ed
 
in
fo
rm
at
io
n?
 R
an
k 
by
 
pu
tt
in
g 
a 
nu
m
be
r 
E
xp
er
ie
nc
e,
 S
A
V
E
R
, 
co
m
m
un
ic
at
io
n,
 o
th
er
 
3 
T
ea
m
 m
em
be
r?s
 
SA
 3
 r
eq
ui
re
m
en
t 
W
ha
t h
as
 y
ou
r 
te
am
 m
em
be
r 
us
ed
 to
 p
re
di
ct
 th
e 
ts
un
am
i 
ge
ne
ra
tio
n?
 R
an
k 
by
 p
ut
tin
g 
a 
nu
m
be
r 
E
xp
er
ie
nc
e,
 S
A
V
E
R
, 
co
m
m
un
ic
at
io
n,
 o
th
er
 
187 
 
Therefore, this approach is error prone, and we extended it by objective SA 
measurements observing the direct performance of users to see what they themselves 
know and what they know about their team members. Likewise, beliefs about the team 
members are elicited from direct questions about the team member?s SA. We believe 
that by using the objective approach we can get a more accurate picture of team SA. 
Table 6.5 shows the example questions for measuring TSA. 
Table: 6.5 Example TSA queries 
TSA 
Level 
Target Questions Possible Answers 
1 TSA 1: Belief about 
team member?s 
cognition 
What do you think your team member 
answered about magnitude, depth and location 
of earthquake? 
8.1 
28km 
15.0 W 115.0 N 
1 TSA 1: Team 
member?s belief 
about own belief 
What do you think your team member believed 
about your answers to magnitude, depth and 
location of earthquake? 
8.1 
28km 
15.0 W 115.0 N 
2 TSA 2: Belief about 
team member?s 
cognition 
What do you believe about how your team 
member will rate magnitude  
High enough, Very high 
 
2 TSA 2: Team 
member?s belief 
about own belief 
What do you think your team member believed 
about your rating of magnitude? 
High Enough. Very 
high 
2 TSA 2: Belief about 
team member?s 
cognition 
According to your belief, what is your team 
member?s understanding of source depth and 
earthquake source location to generate a 
tsunami? 
Likely, Highly likely 
  
2 TSA 2: Team 
member?s belief 
about own belief 
What do you think your team member believed 
about your understanding of source depth and 
earthquake source location to generate a 
tsunami? 
Likely, Highly likely 
3 TSA 3: Belief about 
team member?s 
cognition 
According to your belief what is your team 
member?s prediction about the likelihood of 
tsunami generation? 
Likely, Highly likely
  
3 TSA 3: Team 
member?s belief 
about own belief 
What do you think your team member?s 
believe about your prediction of tsunami 
generation? 
Likely, Highly likely 
 
188 
 
Questions about own cognition are not included in Table 6.6 since they are covered by 
measuring an individual?s SA in Chapter 5.  
For measurement of TSA according to the Salas et al. (1995) definition, answers of each 
team member to SA measurement questions used in Chapter 5 are compared with the 
team member?s answers and one point is awarded to correct matching answers 
otherwise 0 is given. To measure TSA according to Endsley and Jones (2001) definition 
the SA measures of both team members? (Chapter 5) are aggregated.  
6.5.1.2 Participants 
The detail of participants has been provided in section 5.5.1.4 of Chapter 5. Emergency 
managers were randomly grouped into eight teams of two, as described in Table 6.6.  
Table 6.6 Distribution of participants in teams 
Participant 
(Identifier) 
Team 
(Identifier) 
Participant 
Team 
Identifier  
P1 T1 P1T1 
P2 T1 P2T1 
P3 T2 P3T2 
P4 T2 P4T2 
P5 T3 P5T3 
P6 T3 P6T3 
P7 T4 P7T4 
P8 T4 P8T4 
P9 T5 P9T5 
P10 T5 P10T5 
P11 T6 P11T6 
P12 T6 P12T7 
P13 T7 P13T7 
P14 T7 P14T7 
P15 T8 P15T8 
P16 T8 P16T8 
Participants are assigned unique identifiers for easy identification by combining their 
participant and team identifiers as shown. 
6.5.2 Experiment results 
6.5.2.1 Data analysis method  
Figure 6.12 illustrates the data analysis method for SSA data. As in previous analyses, 
marks of 0 and 1 are respectively assigned to incorrect and correct answer to queries. 
Thus, the marks for example questions 1, 2 and 3 targeted at SSA Level 1 are added to 
generate the SSA Level 1 score of participant 1. Similarly, questions 4 and 5 are combined 
to generate an SSA Level 2 score and questions 6 and 7 are combined to generate an SSA 
189 
 
Level 3 score for participants 2 and 3 respectively. An overall SSA score for each 
participant is created by aggregating the marks for all questions targeted at SSA, e.g. the 
SSA score for participant 4 in Figure 6.12 is the sum of the combined scores for questions 
1 to 7.  
To generate a team?s SSA, the scores of team members are averaged. For example, for the 
data in Figure 6.12, team B?s SSA score is the average of the scores for participants 3 and 4. 
All scores calculated as described can be converted to percentages by dividing each 
score by the total number of questions asked at the appropriate level, or asked in total 
for a team, and multiplying by 100.  
SSA 
Level 
Participant 1 
Team A 
Participant 2 
Team A 
Participant 3 
Team B 
Participant 4 
Team B 
SSA1 Question 1 Question 1 Question 1 Question 1 
SSA1 Question 2 Question 2 Question 2 Question 2 
SSA1 Question 3 Question 3 Question 3 Question 3 
SSA2 Question 4 Question 4 Question 4 Question 4 
SSA2 Question 5 Question 5 Question 5 Question 5 
SSA3 Question 6 Question 6 Question 6 Question 6 
SSA3 Question 7 Question 7 Question 7 Question 7 
 
 
 
Figure 6.12 Distribution of example questions and result analysis method 
Similar measurements and analyses are made for TSA.  
For the belief model of Shu and Furuta (2005), TSA questions are combined according 
to the TSA levels and averaged to generate the score for each individual as shown for 
SSA in Figure 6.12. A Team?s TSA score for this model is the average of both team 
members? overall TSA score similar to the SSA example or participant 4 in Figure 6.12.  
SSA 1 score of 
Participant 1 
SSA 2 score of 
Participant 2 
SSA 3 score of 
Participant 3 
SSA score of 
Participant 4 
190 
 
Figure 6.13 shows an example for the belief model. In Figure 6.13 the answer to 
question Q2 for Member A will be considered correct only if it matches the answer of 
Q1 for member B and vice versa.  
 
 
 
 
 
 
 
Figure 6.13 Shu and Furuta (2005)?s belief model 
The common understanding of the situation is measured by comparing the answers 
used in the measurement of the individual?s SA in Chapter 5. Therefore, the team SA 
score will simply be the average of the similar correct answers. The team SA score 
according to Endsley?s model is generated by combining and averaging the individual 
SA score of the participants.  
6.5.2.2 SSA Measurements with and without SAVER 
The results of the SSA measurements (see Figure 6.14) show the overall SSA scores of 
every individual. Participants who used SAVER performed very well compared with 
those who did not use it. A paired sample T-test showed a significant difference (t = 
5.95, p-value < 0.001) between the scores of each participant without SAVER (M=67.3, 
S.D=12.1) and using SAVER (M=78.5, S.D=9.0).  
 
 
 
 
Member A? answers 
Q1: A?s cognition 
Q2: A?s belief about 
B?s cognition 
Q3: A?s belief about 
B?s belief about A?s 
belief  
Member B? answers 
Q1: B?s cognition 
Q2: B?s belief about 
A?s cognition 
Q3: B?s belief about 
A?s belief about B?s 
belief 
191 
 
  
 
                                                                                                                         
 
 
 
 
 
 
Figure 6.14 SSA measures with and without SAVER 
Results are further broken down into the three levels of SSA. Figure 6.15 presents the 
Level 1 SSA results for each individual. 
 
Figure 6.15 SSA Level 1 measures with and without SAVER 
The result of measuring Level 1 SSA using and without using SAVER shows an 
improved performance by those who were using SAVER during the experiment. A 
paired sample T-test shows a significant difference (t = 5.50, p-value < 0.001) between 
the scores of participants using SAVER (M=93.6, S.D=6.5) and for individuals not 
using SAVER (M=84.6, S.D=8.9).  
50
60
70
80
90
100
110
P1
T
1
P2
T
1
P5
T
3
P6
T
3
P9
T
5
P1
0T
5
P1
3T
7
P1
4T
7
P7
T
4
P8
T
4
P3
T
2
P4
T
2
P1
1T
6
P1
2T
6
P1
5T
8
P1
6T
8
SS
A
 le
ve
l 1
 p
er
fo
rm
an
ce
 %
 
Participants 
SSA Level 1  
Using SAVER
Without SAVER
20
30
40
50
60
70
80
90
100
110
P1
T
1
P2
T
1
P5
T
3
P6
T
3
P9
T
5
P1
0T
5
P1
3T
7
P1
4T
7
P7
T
4
P8
T
4
P3
T
2
P4
T
2
P1
1T
6
P1
2T
6
P1
5T
8
P1
6T
8
SS
A
 P
er
of
rm
an
ce
 %
 
Participants 
SSA Measure 
Using SAVER
Without SAVER
192 
 
Figure 6.16 presents the Level 2 results for individuals.  
 
Figure. 6.16 SSA Level 2 measures with and without SAVER 
Figure 6.16 shows the results of measuring Level 2 SSA with and without SAVER. The 
results show that use of SAVER also improves SSA Level 2. A paired sample T-test 
shows a significant difference (t = 7.5, p-value < 0.001) in the score of the participants 
not using SAVER (M=58.3, S.D=17.9) and the score for using SAVER (M=70.3, 
S.D=14.2).  
Figure 6.17 presents the Level 3 SA results for individuals.  
 
Figure 6.17 SSA Level 3 measures with and without SAVER 
10
20
30
40
50
60
70
80
90
100
110
P1
T
1
P2
T
1
P5
T
3
P6
T
3
P9
T
5
P1
0T
5
P1
3T
7
P1
4T
7
P7
T
4
P8
T
4
P3
T
2
P4
T
2
P1
1T
6
P1
2T
6
P1
5T
8
P1
6T
8
SS
A
 le
ve
l 2
 P
er
fo
rm
an
ce
 %
 
Participants 
SSA Level  2 
Using SAVER
Without SAVER
0
20
40
60
80
100
120
P1
T
1
P2
T
1
P5
T
3
P6
T
3
P9
T
5
P1
0T
5
P1
3T
7
P1
4T
7
P7
T
4
P8
T
4
P3
T
2
P4
T
2
P1
1T
6
P1
2T
6
P1
5T
8
P1
6T
8
SS
A
 le
ve
l 3
 P
er
fo
rm
an
ce
 %
 
Participants 
SSA Level 3 
Using SAVER
Without SAVER
193 
 
Figure 6.17 shows the results of measuring SSA level 3 with and without SAVER. The 
results show that SAVER improves SSA at Level 3. A paired sample T-test confirms a 
significant difference (t = 5.17, p-value < 0.001) in the score of participants not using 
SAVER (M=68.7, S.D=14.1) and the score of participants using SAVER (M=84.3 
S.D=13.2).  
Figure 6.18 below shows the overall team?s SSA with and without SAVER by 
aggregating and averaging the team member?s scores. The results show that the 
combined performance of each team is much better when they were using SAVER than 
when they were not. The paired sample T-test shows a significant difference (t = 5.07, 
p-value = 0.001) found in the score of the participants not using SAVER (M=67.3, 
S.D=10.31) and the score of the participants using SAVER (M=78.5, S.D=6.9).  
 
Figure 6.18 Team SSA score with and without SAVER 
6.5.2.3 TSA Measurement 
TSA as ?common SA? measure: Considering the TSA as the degree to which team 
members share a similar understanding of the situation is measured by comparing their 
answers. The team whose correct answers match will get 1 and others get 0. Hence, the 
percentage of correct similar answers developed the common SA or TSA measure. The 
result of the TSA measurement for teams is shown in Figure 6.19. Substantial 
improvement in the score of a team that used SAVER during the experiments can be 
seen compared to those who did not. A paired sample T?test shows that there is a 
30
40
50
60
70
80
90
100
T1 T2 T3 T4 T5 T6 T7 T8
T
ea
m
 S
SA
 p
er
fo
rm
an
ce
 %
 
Teams 
Overall Team SSA 
Using SAVER
Without SAVER
194 
 
significant difference (t = 4.50, p-value = 0.003) in the scores for not using SAVER 
(M=59.5, SD=12.3) and using SAVER (M=81.9, D=8.6). 
 
Figure 6.19 Overall TSA (as common SA) score with and without SAVER 
Results are further broken down into the three levels of TSA. Figure 6.20 presents the 
results for Level 1 TSA. 
 
Figure 6.20 TSA (as common SA) Level 1 measures with and without SAVER 
The result of measuring Level 1 TSA using and without using SAVER shows the team 
using SAVER has performed much better. A paired sample T-test shows that a 
0
10
20
30
40
50
60
70
80
90
100
T1 T2 T3 T4 T5 T6 T7 T8
O
ve
ra
ll 
T
SA
 %
 o
f c
om
m
on
 S
A
 
Teams  
Overall TSA as common SA  
TSA Using SAVER
TSA Without SAVER
30
40
50
60
70
80
90
T1 T2 T3 T4 T5 T6 T7 T8
T
SA
 L
ev
el
 1
  %
 o
f c
om
m
on
 S
A
 
Axis Title 
TSA Level 1 as "Common SA" measure 
TSA1 Using SAVER
TSA1 Without SAVER
195 
 
significant difference (t = 2.36, p-value = 0.05) in the score of the participants using 
SAVER (M=75.0, S.D=10.1) and the score without using SAVER (M=60.7, S.D=14.8). 
Figure 6.21 below presents the Level 2 TSA data for the teams.  
 
Figure 6.21 TSA (as common SA) Level 2 measures with and without SAVER 
Figure 6.21 shows the results of measuring Level 2 TSA with and without SAVER. The 
results show that TSA level 2 of teams using SAVER is 100%. A paired sample T-test 
shows a significant difference (t = 9.00, p-value < 0.001) in the score of the participants 
not using SAVER (M=71.9, S.D=3.1) and the score for using SAVER (M=100.0, 
S.D=0.0).  
Figure 6.22 below presents level 3 TSA for the teams.  
 
Figure 6.22 TSA (as common SA) Level 3 measures with and without SAVER 
0
20
40
60
80
100
120
T1 T2 T3 T4 T5 T6 T7 T8T
SA
 L
ev
el
 3
 %
 o
f c
om
m
on
 S
A
 
Teams 
TSA Level 2 as common SA measure 
TSA2 Using SAVER
TSA2 Without SAVER
0
20
40
60
80
100
120
T1 T2 T3 T4 T5 T6 T7 T8T
SA
 L
ev
el
 3
 %
 o
f c
om
m
on
 S
A
 
Teams 
TSA Level 3 as common SA measure 
TSA3 Using SAVER
TSA3 Without SAVER
196 
 
Figure 6.22 shows the results of measuring Level 3 TSA with and without SAVER. The 
result again shows that SAVER also improves the TSA at Level 3. A paired sample T-
test shows a significant difference (t = 2.04, p-value = 0.08) in the score of the 
participants not using SAVER (M=45.8, S.D=24.8) and the score of the participants 
using SAVER (M=70.8, S.D=21.3).  
TSA Measure for cognition and belief model: To measure TSA according to the 
cognition and belief model, the method proposed by (Nonose et al., 2010) is used. 
Individual responses to the questions about their own cognition, beliefs and their team 
members? beliefs are compared as shown in Figure 6.15 above. When the cognition 
matches the belief of a team member, 1 point is awarded to the individual for that 
question. The percentage of the correct matches is calculated to develop the TSA 
measure of each individual.  
The result of TSA measurement using this cognition and belief model is shown in 
Figure 6.23. The TSA performance percentage of the individuals using SAVER is 100%, 
which means that they were able to correctly state their team members? cognition for all 
the questions. A paired sample T?test shows that there is a significant difference (t = 
3.47, p-value = 0.003) in the score for not using SAVER (M=84.4, S.D=17.8) and the 
score for using SAVER (M=100.0, S.D=0.0).  
 
Figure 6.23 TSA score according to Shu and Furuta (2005) model  
TSA Measure (Endsley?s Model): To measure TSA according to Endsley?s model, 
team members? individual SA performance score is averaged. The result of TSA 
0
20
40
60
80
100
120
P1
T
1
P2
T
1
P5
T
3
P6
T
3
P9
T
5
P1
0T
5
P1
3T
7
P1
4T
7
P7
T
4
P8
T
4
P3
T
2
P4
T
2
P1
1T
6
P1
2T
6
P1
5T
8
P1
6T
8
T
SA
 P
er
fo
rm
an
ce
 %
 
Participants 
TSA Measure (Cognition & Belief  Model) 
TSA Using SAVER
TSA Without SAVER
197 
 
measurements according to Endsley?s model is shown in Figure 6.24. The result shows 
that the TSA performance of the teams using SAVER is much better than those not 
using SAVER. A paired sample T?test shows a significant difference (t = 4.42, p-value 
= 0.003) in the score of non-SAVER users (M=83.4, S.D=7.2) and the score of SAVER 
users (M=91.8, S.D=3.1). 
 
Figure 6.24 TSA score with and without SAVER 
The above results are further broken down into the three levels of TSA to analyse the 
TSA performance of teams at each level. Responses to questions targeted at each level 
of TSA are isolated according to the TSA level and the percentage score of the team 
members is averaged to provide the team?s TSA at the requisite TSA level.  
Figure 6.25 presents level 1 TSA for the teams. 
50
60
70
80
90
100
T1 T2 T3 T4 T5 T6 T7 T8O
ve
ra
ll 
T
SA
 P
er
fo
rm
an
ce
 %
 
Team 
Overall TSA Measure (Endsley's model) 
TSA Using SAVER
TSA Without SAVER
198 
 
 
Figure 6.25 TSA Level 1 measures with and without SAVER 
The result of measuring Level 1 SSA using and without using SAVER shows an 
enhanced performance by the participants using SAVER during the experiment. A 
paired sample T-test shows a significant difference (t = 4.26, p-value = 0.004) in the 
score of the participants using SAVER (M=87.3, S.D=5.3) and the score while not 
using SAVER (M=79.4, S.D=6.0).  
Figure 6.26 presents Level 2 TSA for the teams.  
Figure 6.26 shows the results of measuring Level 2 TSA with and without SAVER. The 
results show that the TSA of the SAVER users is 100%. A paired sample T-test shows a 
 
Figure 6.26 TSA Level 2 measures with and without SAVER 
50
55
60
65
70
75
80
85
90
95
100
T1 T2 T3 T4 T5 T6 T7 T8
T
SA
 L
ev
el
 1
 P
er
fo
rm
an
ce
 %
 
Teams 
TSA Level 1 (Endsley's model) 
TSA1 Using SAVER
TSA1 Without SAVER
0
20
40
60
80
100
120
T1 T2 T3 T4 T5 T6 T7 T8T
SA
 L
ev
el
 2
 P
er
fo
rm
an
ce
 %
 
Teams 
TSA Level 2 (Endsley's model) 
TSA2 Using SAVER
TSA2 Without SAVER
199 
 
significant difference (t = 5.22, p-value = 0.001) in the score of the participants not 
using SAVER (M = 82.8, S.D = 9.3) and the score of the participants using SAVER 
(M=100.0, S.D=0.0).  
Figure 6.27 below presents level 3 TSA for the teams.  
 
Figure 6.27 TSA Level 3 measures with and without SAVER 
Figure 6.27 shows the TSA Level 3 score of participants with and without SAVER. The 
results again confirm that SAVER improves TSA at Level 3. A paired sample T-test 
shows a significant difference (t = 3.75, p-value = 0.007) is found in the score of the 
participants not using SAVER (M=78.7, S.D=11.9) and the score of the participants 
using SAVER (M=94.6, S.D=6.4).  
The results above show that TSA on all three levels was improved by using SAVER 
during the experiments. Since TSA score was based on the individual SA scores and 
significant improvement in these scores have been discussed earlier in Chapter 5, 
noteworthy improvements in TSA are expected and confirmed. 
6.6 Summary and Conclusions 
In mass evacuation operations, more than one organization is involved in decision 
making and task completion and the huge involvement of many individuals from 
numerous organisations highlights the importance of co-ordinated efforts. Hence, in 
addition to the individual SA, the team SA and shared SA are also critical. This chapter 
examines the requirement of team co-ordination to develop a necessary level of shared 
SA and TSA.  
0
20
40
60
80
100
120
T1 T2 T3 T4 T5 T6 T7 T8
T
SA
 L
ev
el
 3
 P
er
fo
rm
an
ce
 %
  
Teams 
TSA Level 3 (Endsley's model) 
TSA3 Using SAVER
TSA3 Without SAVER
200 
 
Developing a common understanding of a prevailing situation is the first step in making 
a decision to normalise the situation. Getting a consensus is time consuming, especially 
when team members have varied roles with different backgrounds and diverse 
responsibilities. Moreover, large collaborative tasks require the division of work in 
teams, where team members require support from others, mostly in the form of 
information. To support the team members effectively and efficiently, information 
required by the other team members can be shared proactively. However, since many 
organisations are involved in large operations, it is difficult for an individual to 
remember the details of everyone?s requirements who are dependent on them.  
In this chapter, an information system design based on proactive sharing of information 
using ontologies is proposed and evaluated. Information requirements and user context 
ontologies developed in Chapter 4 and Chapter 5 respectively are re-used and extended 
by adding the source and destination of the information flows. This enables the 
automatic routing of the right information requirements to the right people using an 
ontology reasoner (Fact ++).  
It is proposed that providing an explicit assessment of the situation improves the 
common awareness among team members. Sharing the team member?s understanding 
of the situation also improves their shared understanding. Moreover, automatic 
reminders about the team member?s requirements improves SSA and also lifts the 
cognitive load of emergency managers since they do not have to remember every team 
member?s information requirements.  
The proposed design extends the prototype, SAVER, based on the above ideas. To 
evaluate this extension, the SSA and TSA of emergency managers are measured at all 
three levels. The results show that SAVER not only significantly improves the SSA at all 
three levels but also improves the overall SSA of teams. It appears that the SAVER?s 
suggestions about the information requirements of team members to develop and 
maintain SA improves the SSA of the teams. Reminding them about their own 
requirements and their team members, helps them to keep the team member?s 
requirements in sight. Moreover, this facility reduces the cognitive load needed to 
remember team member requirements since SAVER keeps track of these and issues 
reminders when required. 
201 
 
TSA has been measured according to its three prevalent definitions (models). Firstly, 
TSA is measured according to Endsley?s model (Endsley & Jones, 2001), which 
describes TSA as the degree to which every team member possesses the required SA for 
his/her responsibilities. For this model, the individual SA of team members is combined 
to generate a TSA measure. Secondly, TSA is measured as a common or shared 
understanding of the situation (Salas et al., 1995), individual answers from the individual 
SA measure are compared to see the matching answers. Lastly, TSA is measured 
according to Shu and Furuta (2005) model of cognition and belief. For all three 
prevalent models of TSA, performance of the SAVER users was expressively better 
than the non-SAVER users. SAVER improved TSA by following a few simple steps like 
sharing the SA of one team member with the others, or by providing the same explicit 
assessment of the situation to all members. These steps not only helped team members 
to develop a common understanding of a situation, but also helped to support them to 
develop their beliefs about the team members? SA.  
In our experimental environment teams consisted of only two members and the 
constrained design, implementation and evaluation of SAVER must be seen as a proof-
of-concept study. In real emergencies, team sizes would be much bigger and managers? 
roles and situations more complex. Scaling SAVER could produce a system with the 
capability of improving SSA and TSA resulting in highly effective and efficient team 
decision making and collaborative task completion. This functionality is at the core of 
mass evacuation operations and overall emergency management and the SAVER 
concept has much to offer as a building block of change. 
 
 
 
 
 
 
 
202 
 
 
DISCUSSION  
7.1 Introduction  
The requirements of emergency managers to act effectively and efficiently in emergency 
situations are highly dynamic and complex. Both physical and cognitive factors affect 
their decisions and activities. This chapter first identifies situation awareness (SA), 
shared SA (SSA) and team SA (TSA) as the factors that affect individual and team 
decision making and task performance in emergency situations, and proposes changes 
to their formal definitions to clarify their meanings. The main part of the discussion 
then focuses on an interpretation of the results of the SAVER design, implementation 
and evaluation studies to improve these factors as set out in previous chapters. The 
chapter then discusses the relationships between the various models of TSA and 
describes the implementation requirements of an operational SAVER system. The 
chapter concludes with a summary of the whole research.  
7.2 Formalising SA, SSA and TSA Definitions 
As mentioned earlier (Chapter 2), there are multiple definitions of SA, SSA and TSA 
(Uhlarik & Comerford, 2002), not all of which are consistent, and a challenging task 
during this research was to understand and select the right ones since these concepts are 
central to the study. The following discussion, whilst somewhat arcane, endeavours to 
unravel and clarify some of the definitions and demonstrate the rationale behind 
choices. The presentation repeats some of the material presented in the literature review 
(Chapter 2), in order to develop a consistent and coherent approach to SA. 
Situation awareness is the most fundamental concept since it underpins the other two 
(SSA and TSA), and accordingly it has been the subject of the highest level of 
controversy. Leaving aside vague, everyday definitions that lack scientific rigour and 
domain specific meanings that defy generalisation,. a common assumption in several 
definitions is that a certain level of SA may be enough to carry out a task successfully 
(assuming adequate execution) without acknowledging that a higher level of SA can lead 
to improved performance. Similarly, there is insufficient recognition in several 
203 
 
definitions that SA is a dynamic factor, valid for a particular time and that it may need 
updating when the situation changes. These aspects are especially important in 
emergency situations.  
The most widely used definition of SA by Endsley (1995a), as: 
?The perception of the elements in the environment within a volume of time and space, the comprehension 
and the projection of their status in near future? (Endsley, 1988a).  
has also faced criticism. An interesting debate concerns whether SA is a process or a 
product. Endsley?s definition categorises SA as a product that is the result of a process 
of ?situation assessment?. However, others like (Fracker, 1991; Uhlarik & Comerford, 
2002) call it a process to obtain the required knowledge of a situation. They argue that 
the perception, comprehension and projection elements of SA definition can all be 
considered as processes.  
Since SA is a measurable outcome characterised by absolute (often numerical, e.g. an 
aircraft?s altitude) or relative quantities, Endsley?s definition seems the more logical. 
Another issue (Uhlarik & Comerford, 2002) concerns whether measures of SA are 
objective or subjective quantities. Endsley believes they are objective since the measured 
values can be compared with true values whereas Fracker (1991) prefers a subjective 
interpretation because data is acquired via self-reports rather than direct observation. 
Although this is perhaps a minor distinction, it has led Fracker to categorise SA data as 
retrospective (obtained on completion of a task), concurrent (on-line acquisition) and 
frozen (collected at a pause in the task). The freeze technique resolves the 
objective/subjective debate to some extent because, as Endsley argues, pausing a 
process allows participants to focus on task related questions and provide considered 
answers without the distraction of time related problems and mental overload issues. 
These arguments, and its simplicity, recommend Endsley?s model of SA as a theoretical 
framework and the freeze method of measurement for the current research.  
However, an even more important reason for accepting her definition is the valuable 
division of SA into three hierarchical levels, which allows the construct to be measured 
easily and effectively. Moreover, this division allows the abstraction of SA requirements 
and the development of design guidelines and training strategies to support the 
acquisition of SA at different levels. In addition, Endsley?s model attempts to outline the 
204 
 
various factors that affect the development and maintenance of SA and includes a 
comprehensive set of individual, task and system factors (Salmon et al., 2009). Finally, 
Endsley (1988b; 1990; 2000) provides a proven method, Situation Awareness Global 
Assessment Technique (SAGAT), which measures SA at different levels in terms of the 
number of correct answers to questions asked about the situation. Therefore, rather 
than a product or process, SA can be regarded as a state of knowledge about the 
situation, which is measured as a ?degree? or ?extent?; the higher the degree the better 
the SA. We therefore propose a slight modification to the Endsley?s SA definition, 
which may silence some of the debates about the concept:  
?The degree to which an individual has the perception of the elements in the environment within a 
volume of time and space, the comprehension, and the projection of their status in the near future?. 
The rather odd sounding ?volume and space? component in the definition could also be 
replaced to generalise the change aspect as: 
?The degree to which an individual has the perception of the elements in the environment at a particular 
instant, the comprehension and the projection of their status in the near future? 
One more reason for considering SA as a degree (extent) is the connection with 
Endsley?s definitions of SSA and TSA. Since SSA and TSA are similar concepts as SA 
and are based on it, they are also defined in terms of degree. For example, the most 
widely used and accepted definition of SSA is  
?The degree to which team members have the same SA on shared SA requirements? (Endsley & 
Jones, 2001). 
 However, this definition assumes that the shared SA is always correct whereas in 
practice it could be based on wrong or incomplete SA requirements. In the current 
work, our evaluation of SAVER avoids this ambiguity by measuring not only shared 
awareness on SA requirements but also by checking the requirements for accuracy. In 
addition, the definition does not indicate the required degree of the shared awareness 
between participants, which needs only to extend to those requirements necessary for 
successful task completion. Requirements that are irrelevant to a shared awareness of a 
particular task will not contribute to the SSA for that task so that a complete (or same) 
shared awareness between co-operating team members is not essential. We can handle 
this condition by making a small change to the definition: 
205 
 
 ?The degree to which every team member has the awareness of his/her relevant team member?s 
requirements of perception, comprehension and projection of the situation?.  
Another important concept related to the collaborative tasks is team SA (TSA). Like SA, 
and SSA, TSA has also faced conflicting definitions (Endsley and Jones, 2001; Shu & 
Furuta, 2005; Salas et al., 1995). Endsley and Jones defined TSA as: 
The degree to which every team member possesses the situation awareness required for his or her 
responsibilities? (Endsley, 1995b; Endsley & Jones, 2001). 
According to this definition, TSA is the accumulated SA of all team members, a 
statement that disguises the interactions between team members that update each 
other?s SA and influence their task performance by supporting them and dividing the 
workload. These important team factors should be evident within the definition. 
Salas et al. (1995) gave a totally different perspective to TSA as: 
The shared understanding of the situation among team members at one point in time?.  
focusing on the degree to which team consensus about the situation and its future 
evolution are required for decision making and collaborative task execution. Without 
consensus a team cannot act co-operatively to normalise a current or forecast situation. 
As with SSA, the definition of TSA by Salas et al. (1995) does not say anything about 
the correctness of the shared understanding of the situation. However, in our evaluation 
of SAVER while measuring TSA, we considered only those shared answers of team 
members that were correct in the first place.  
Close to the ideas of Salas et al. (1995) on TSA is the cognition and belief model by Shu 
and Furuta (2005). According to these workers, TSA is defined as  
?Two or more individuals who share the common environment, up to the moment understanding of the 
situation of the environment and other person?s interaction with the co-operative task.? (Shu & Furuta, 
2005). 
 In addition to ?sharing the understanding of the situation?, this definition adds the 
team member?s interaction with the co-operative tasks missing from the Endsley 
version. This is useful for collaboration since to provide timely input and support to a 
team member, an individual should know when that team person is taking action. 
206 
 
Moreover, Shu and Furuta also introduced the concepts of mutual beliefs which 
describe what each individual believes about his/her team member?s understanding of 
the situation, and beliefs about what that team member believes about the individual?s 
understanding. Having a correct view of a team member?s understanding of the 
situation is very useful since, in this way, a team member?s SA can be updated if there is 
some problem in the understanding of that team member. Therefore, these concepts of 
cognition and belief are critical for the team?s collaborative tasks and team decision 
making. Although for the evaluation of SAVER we have measured TSA according to all 
the models discussed above, we consider the cognition and belief model of Shu and 
Furuta (2005) as the most realistic, since it encompasses the TSA definition by Salas et 
al. (1995). However, the belief about a team member?s understanding should be part of 
the definition and we propose that TSA is:  
?The degree to which related team members have the shared perception, comprehension and projection of 
the situation and correct belief about a relevant team member?s perception, comprehension and projection 
of the situation? 
In this definition the term ?related team members? limits the sharing requirement to 
only those team members who need to have a similar understanding of the situation 
since some may not require to understand the situation at all, e.g. if it is not relevant for 
their task or job role. Moreover, the ?relevant team member? in the second half of the 
definition is the person whose belief needs to be known by the individual to ensure 
effective collaboration.  
The new definitions proposed above attempt to clarify the relationships between the 
definitions of SA, SSA and TSA. More work is needed to produce a fully integrated and 
consistent perspective but the suggested improvements help the design of experiments 
to measure the different forms of situation awareness. Viewing SA as a state of 
knowledge about the situation by an individual or a team also underlines the utility of 
computer based systems to improve awareness.  
Moreover, to support SSA and TSA from every aspect, SSA and TSA are divided into 
three levels like SA (Endsley, 1995b). The division of SSA and TSA into levels will help 
in identifying the detailed information requirements of each level. See Table 7.1 for the 
division of SSA and TSA into three levels. This division has not been reported 
previously and it would be interesting to explore it further.  
207 
 
Table 7.1 Levels of SSA and TSA 
Concept Level Definition 
SSA 1 The degree to which every team member has the awareness of his/her 
relevant team member?s requirements of perception of the situation 
SSA 2 The degree to which every team member has the awareness of his/her 
relevant team member?s requirements of comprehension of the situation 
SSA 3 The degree to which every team member has the awareness of his/her 
relevant team member?s requirements of projection of the situation 
TSA 1 Degree to which relevant team members possess common/shared knowledge 
of the relevant elements (perception) to the prevailing situation and correct 
beliefs about the relevant team member?s perception 
TSA 2 Degree to which relevant team members possess common/shared knowledge 
on the understanding (comprehension) of the situation elements and correct 
beliefs about the relevant team member?s comprehension 
TSA 3 Degree to which relevant team members possess common/shared knowledge 
on the projection of the situation elements and correct beliefs about the 
relevant team member?s projection 
 
7.3 Information Requirements for SAVER Design 
The objective of this study is to develop a computer based system to support the SA of 
individuals and teams, i.e. SA, SSA and TSA. The systems design is therefore based on 
the information required for developing and maintaining the different forms of 
awareness. Obtaining these information requirements is complex since they include not 
only the factual information needed for the various roles and associated tasks but also 
information on the cognitive processes that lead to the realisation of situation 
awareness.  
This study has used a cognitive task analysis technique named Extended Goal Directed 
Information Analysis (EGDIA), which has proved very useful in capturing both 
physical and cognitive information requirements (section 4.4.1.3, Chapter 4). The scope 
of the study was limited by the number of experts (emergency managers) and their 
availability for the simulation experiments and consequently, the requirements analysis 
was restricted to the roles of these participants. However, 13 out of the 16 New Zealand 
Civil Defence Emergency Management Groups participated and this number was 
sufficient to cover the majority of the important roles and validate their requirements.  
208 
 
As described in earlier chapters, SAVER was evaluated by a tsunami simulation so that 
the information requirements covered the roles for mass evacuation following such an 
event. These requirements can be adapted and extended for other types of emergency, 
e.g. flood, hurricane, volcanic eruption and a production version of SAVER could use 
EGDIA to develop and validate comprehensive sets of information requirements for 
each situation.  
In an emergency, responders will invoke different mental models depending on their 
experience and they can also use different data sets to reach the same conclusion. This 
study therefore, only attempts to gather the standard or generic information 
requirements sufficient to develop and maintain the levels of SA needed for decision-
making and task performance. However, the information requirements were captured 
via questionnaires from managers throughout New Zealand and the level of consensus 
across the responses strongly suggests that the generic set was detailed enough to 
achieve these goals. Again, in a production environment, the SAVER design could 
update data sets as dictated by improvements in the situation awareness.  
Figure 7.1 (next page) summarises the relationship between the various levels and types 
of information requirements used in this research and their involvement with decision 
making and task performance.  
7.4 Ontology-based SA, SSA and TSA 
Several researchers (Mathues et al., 2003; McGuinness et al., 2003; Horney et al., 2003; 
Boury-Brisset, 2003) have proposed ontologies for the representation of situation 
awareness. These studies use mainly general descriptions to describe classes and their 
properties and demonstrate how a situation can be expressed in computer processable 
terms. The present study takes the approach well beyond this purely modelling, 
descriptive stage and maps every level of SA to ontologies with the goal of improving 
situation awareness at each level. Coupling ontologies with a reasoning capability also 
leads to projections (Level 3 SA) that generate additional knowledge about the concepts 
not explicitly described in the relationships.  
 
 
 
209 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 7.1 Various types and flows of information requirements  
As explained in Chapter 4 the information requirements of emergency managers are 
represented in Web Ontology Language, OWL, which is machine readable. In this 
research, OWL reasoning capabilities have proven useful for the simulated scenario 
experiments and it is evident that the expressive power of OWL is sufficient to model 
Team SA 
Decision-making 
Individual SA 
Shared SA 
Collaborative 
Goals & Tasks 
  
Information required for 
What are Tasks? 
How to perform the tasks? 
What are Goals? 
Situation Awareness 
  
Level 1 requirements 
Level 2 requirements 
Level 3 requirements 
Team SA 
  
What is team members? SA? 
Who are team members? 
Shard SA 
  
When to fulfil requirements? 
Who are relevant team 
Team members? SA 
Resources 
  
Supplies in welfare camps 
Information on Human 
Information on evacuation 
210 
 
even very complex situations. This allows the development of an architecture that 
provides a detailed approach and procedures, not just for modelling situations, but for 
automatic situation assessment, which is a principal objective of the study. Another 
advantage arises from the re-usability of modelled information requirements and the 
user?s context information as seen later in the study when ontologies initially developed 
to support individual SA are re-used for supporting SSA and TSA. Moreover, after the 
dynamic modification of relationships or addition of concepts, the ontology reasoner 
(e.g. Fact Plus Plus) can check for errors and contradictions so that the modification can 
be automatically validated. Last, but not least, ontologies can be used to directly 
represent the reality of a situation without introducing abstractions like database tables, 
dictionaries or documents.  
However, all these advantages come at some cost. Developing ontologies is difficult and 
time consuming while re-usability of the existing ontologies has issues related to the 
level of abstraction and the intent of their creation. Making them re-usable takes a 
considerable amount of time and effort and it can be more straightforward to develop a 
new ontology from scratch. Nevertheless, substantial research is under way to improve 
the re-usability of ontologies. In addition, current ontology reasoners (Racer, Pellet, Fact 
Plus Plus, Hermit, etc.) still have some performance issues when classifying and 
reasoning for complex and large OWL ontologies (Dentler et a., 2011; Huang et al., 
2008). Though the rapid increase in computing power is gradually overcoming this 
problem, it is still an issue now and in the foreseeable future.  
On a positive note, the reasoning and scalability capabilities of OWL are improving due to 
the addition of new features such as property chains, richer data types and data ranges, 
qualified cardinality restrictions and asymmetric, reflexive, and disjoint properties 
(Bassiliades, 2012). Mapping and integrating OWL and database technologies can also 
improve performance and extract more information from the data, (Zhuge et al., 2008). 
The use of ontologies in various parts of this project is summarized in Figure 7.2. 
 
 
 
 
211 
 
 
 
 
 
 
 
Figure 7.2 Various ontologies in SAVER 
7.5 Improving SA using SAVER 
As the evaluation results show, the use of SAVER improves SA at all three levels. Few 
studies have measured the SA of participants in ?before and after? environments designed 
to improve situation awareness as this study does. Most of the studies have measured SA 
to see the effect of some specific factor, e.g. experience level, communication or training 
etc. (Hauland, 2008; Saner et al., 2009; Bolstad and Endsley, 2003; Bolstad, Cuevas, and 
Costello, 2005). Moreover, to our knowledge, there is no study in the emergency decision 
making domain that attempts to improve SA using computer based systems and then 
evaluates the design by measuring the SA of users. Direct comparison with the SA 
measurement results of other studies is therefore not possible.  
As noted, SAVER uses SA-oriented design principles proposed by Endsley et al. (2003) 
(See also Bolstad et al., 2006) in which relevant Level 2 and Level 3 SA information is 
provided to emergency managers in a format that describes the significance and 
implication for current and future situations so that they can internalise it and process it 
quickly. Only contextualised and personalised information is delivered depending upon 
the managers? roles in the current emergency phase and the tasks they are performing. 
In contrast, when emergency managers receive raw data about the situation, they need 
to select the relevant components and then process them, which is a time consuming 
task under dynamic and stressful conditions. Use of SAVER therefore saves time and 
reduces distracting workload, valuable benefits during a crisis response. The time 
evaluation in the current study only reflects the time for retrieving SA information 
whereas it would be interesting to see the time taken by each user to develop their 
Information 
requirements 
ontology 
Context ontology 
Users, roles, tasks, 
goals ontology 
SAVER Situation 
Ontology 
Team Situation 
awareness 
Individual 
Situation 
Awareness 
Shared Situation 
Awareness 
212 
 
situation awareness after receiving the information. Since SA is a cognitive attribute, 
capturing the time taken to develop SA is quite involved and it will require careful 
experimental design.  
Since emergency managers? perceptions affect their comprehension and the resulting 
projection of the situation, mistakes or errors in Level 1 SA can result in flawed SA at 
Levels 2 and 3. However, though SAVER supports SA on all three levels, its use does 
not require its users to keep up with the SA on every level. For example, if there are 
mistakes in perceiving the situation elements (Level 1 SA), SAVER can directly update a 
user?s SA on Level 2 via the direct interpretation of situation elements. Similarly, even if 
the understanding of the situation (Level 2 SA) is defective, SAVER will update Level 3 
SA directly reducing and controlling mistakes in decision making and task performance.  
Analysis of the results show a substantial difference in the SA performance of the 
novice and experienced emergency managers when the participants were not using 
SAVER (section 5.5.2.4). In this circumstance, the SA performance of the experienced 
emergency managers was significantly better than the novice ones. This agrees with the 
explanation by Endsley, (1995a) and others (Klein, 1999) that the SA of individuals 
depends on their experience level. However, when participants were using SAVER, 
there was no statistically significant difference in the SA of both groups based on their 
experience levels. This is an important conclusion since it reduces the dependence on 
experienced managers who may not be readily available during an emergency. With the 
effective use of such systems, even novice emergency managers can perform as well as 
experienced emergency managers. SAVER also improves the SA of the experienced 
emergency managers, thus, reducing the chances of errors. These results also highlight 
the improvement that SAVER brought to the SA performance of the participants ? see 
Table 7.2 which shows the percentage of correct answers to questions at the freeze 
points of simulations with and without SAVER (section 5.5.2.4).  
 
 
 
 
213 
 
Table 7.2 Comparison of experienced and novice emergency managers? 
performance with and without SAVER 
 
Manager Type 
% Correctly Answered Questions 
Without SAVER With SAVER 
Novice managers 75.2 89.9 
Experienced Managers 84.0 93.8 
 
Analysis of the time for answering the SA questions also illustrates that SAVER users 
took significantly less time than the non-SAVER users. Hence, the efficiency of the 
SAVER users was much better in retrieving SA information on all three SA levels. 
Saving time can significantly improve decision making and task performance.  
The human computer interface is a crucial component of emergency systems design, 
which should be intuitive and simple enough to make the system usable. A complex 
interface would keep the user occupied in figuring out how to perform the task instead 
of concentrating on the solution. The simplicity of presentation and the focus on SA 
requirements contributed directly to the positive assessment of SAVER. Nevertheless, 
the user interfaces can be improved further by performing detailed requirements 
analysis and involving emergency managers in the design of screens that support their 
individual and combined roles and tasks. 
Evaluation of SAVER by only 16 emergency managers was enough for a proof of 
concept but more experiments with more users are required to generalise the findings. 
The current study evaluates SAVER by measuring the SA of the emergency managers 
but the quality of the decisions made or tasks performed could also be used as 
parameters for evaluation and perhaps reveal other important factors affecting SA. 
However, differentiating between ?good? and ?bad? performance is intrinsically subjective 
and difficult to formulate.  
7.6 Improving SSA using SAVER 
Co-ordination and information sharing are central to the collaborative tasks of a team 
and SAVER proactively shares information requirements between team members, 
214 
 
making use of contextual information, the overall common goals, the shared tasks, and 
the resulting flow of the information to fulfil members? information requirements.  
While designing the SSA module of SAVER, it was hypothesised that requesting users 
for the information requirements of their relevant team members would not only make 
them aware of these member?s requirements but also lead to the timely fulfilment of 
their information needs. This hypothesis was tested by measuring the user?s SSA. 
Significant improvements were found in the SSA of SAVER users compared with non-
SAVER users, confirming the successful design of the SSA module.  
The teams for these tests were randomly selected so that members were unaware of 
each other?s background, experience level and working style (tacit knowledge) (Paton & 
Jackson, 2002) etc. However, they still managed to perform very well during the 
collaborative task, strongly suggesting that the proactive sharing of team members? 
information requirements can be very useful in such situations making it unnecessary 
for members to know about these requirements beforehand. In addition, they are not 
required to remember the requirements since SAVER can remind them. These 
advantages become increasingly important as the size or number of collaborating teams 
increases.  
All the participants were aware of the Planning and Intelligence (P&I) function of 
Coordinated Incident Management Systems (CIMS) but not necessarily familiar with the 
preparation of a SitRep. Preparing the SitRep is the responsibility of specialised roles 
namely Situation Reporting and Planning and Intelligence Manager. Only half an hour 
of the introductory session was allocated to the participants describing their 
responsibilities and how to write the SitRep. Although no data about their previous 
knowledge of the task was gathered, their SSA performance was quite satisfactory (see 
section 6.5.2.2) while using SAVER (M=78.5). These results can be credited to SAVER, 
which was reminding the participants about their team member?s relevant information 
requirements as well as compensating for the participant?s lack of specialist knowledge. 
Though the SSA of non-SAVER users is significantly (statistically) lower than the 
performance of SAVER users, it is also reasonably good (M=67.3). This performance 
level is almost certainly due to the ease of the task since there were not many shared 
information requirements or flows in the simulations. It would be interesting to evaluate 
SSA in more complex situations where many shared and uncommon information 
215 
 
requirements are involved, and to obtain a more detailed picture of SSA by collecting 
data about team members? knowledge of the roles, tasks of others using and without 
using SAVER. Furthermore, the current study uses teams of only two members (to 
make eight teams); experiments with larger teams will be useful to produce results that 
are more generalizable.  
7.7 Improving TSA using SAVER  
When teams are involved in decision making, consensus is required to reach a decision. 
Agreement on an understanding of the situation is the first step, discussing the options 
or actions to normalise it comes next. The larger the team, the more time it takes to get 
an agreement. This is because different team members have diverse backgrounds, 
experience, intelligence level etc., and they process the same information differently 
according to their different roles and attendant sub-goals. SAVER provides the explicit 
situation analysis along with the logical arguments (rules it uses to reach conclusion) to 
provide a common understanding of the situation. 
In this study, the hypothesis that providing the explicit situation assessment improves 
the common situation understanding is tested by measuring the TSA of participants. 
This is done by measuring the individual SA of the participants with and without 
SAVER and checking commonality across the team members. The results show that the 
TSA score of SAVER users is significantly better than for the non-SAVER users. This 
result is, of course, intuitively expected but the statistical evidence shows 
uncontrovertibly that SAVER overcomes the ?silo effect? of independent assessment 
and awareness and helps team members to make consensus decisions. Because SAVER 
is a prototype working with comparatively simple scenarios, the conclusion should be 
regarded as proof-of-concept but it implies that the result will be scalable to large 
production systems and more complex situations, and this can be confirmed by more 
studies.  
The current study also evaluated SAVER by measuring team SA according to the TSA 
definition advanced by Endsley and Jones (2001). For this purpose, team member 
individual SA scores were averaged to get the team SA score. The TSA score calculated 
in this way for the teams using SAVER was significantly better than for the non-
SAVER user teams.  
216 
 
The TSA tests show the strong dependency between TSA and individual SA and the 
design of systems supporting TSA should not only concentrate on team communication 
and co-ordination but also on the SA of individuals. Moreover, instead of concentrating 
on one of these TSA models, the system design should consider all three as they 
complement each other and the resulting team performance. If one of them is missing, 
TSA will be flawed. Hence, instead of calling all of them TSA, which creates confusion, 
they can be combined to represent team SA as defined in the previous section.  
Though the tacit knowledge and other team factors like communication between the 
team members, modes of communication, decentralisation of the team etc., are critical 
in team collaboration (Paton et al., 1999; Paton & Jackson, 2002; Smallman & Weir, 
1999), they are not considered in the current study since their impact on the team 
performance is already studied. Moreover, tacit knowledge is acquired by getting team 
members to work together. SAVER?s objective is to minimise this dependability. This 
will allow the more versatile and dynamic team creation. By sharing a team member?s 
awareness directly, team members are not required to know each other to develop 
common understanding or beliefs. However, this hypothesis needs to be tested.  
7.8 Relationship Between Individual SA and Various Models of TSA  
From the results discussed above, it can be seen that the common awareness of a situation 
(Salas et al., 1995) has some role to play in the cognition and belief model (Shu & Furuta, 
2005) since TSA, according to the cognition and belief model, is high for those teams 
whose common SA (TSA) is high. Participants believed that their team members were 
choosing the same options as they were. Similarly, teams having high TSA according to 
the Endsley?s model, have also high common SA. This effect was high when teams were 
using SAVER since they knew they were getting the same situation assessment. While 
using SAVER, the TSA score of participants is 100% due to the correlation between 
cognition, belief and common SA and cannot be compared without using SAVER. 
However, to confirm these relations, results of all the individuals (with and without 
SAVER) are combined and are checked for correlations as shown in Table 7.3.  
 
 
 
217 
 
Table 7.3 Correlations between TSA as combined SA (Endsley & Jones, 2001), 
TSA as common SA (Salas et al., 1995), and TSA as cognition and belief model 
(Shu & Furuta, 2005) 
 
TSA as 
Belief & 
Belief 
TSA as 
Common SA 
TSA 
(Endsley?s 
model) 
Individual 
SA 
TSA as 
Belief & 
Belief 
Pearson 
Correlation 
1 .84 .57 .32 
Sig.   p < 0.001 .001 .077 
     
TSA as 
Common 
SA 
Pearson 
Correlation 
.84 1 .73 .54 
Sig. p < 0.001   p < 0.001 .001 
     
TSA 
(Endsley?s 
model) 
Pearson 
Correlation 
.57 .73 1 .56 
Sig. .001 p < 0.001   .001 
     
Individual 
SA 
Pearson 
Correlation 
.32 .54 .56 1 
Sig. .077 .001 .001   
     
 
Pearson correlation tests show that all the results in Table 7.3 are significant. Individual 
SA is highly correlated with Endsley?s model of TSA since this form of TSA is simply 
the aggregation of a team member?s individual SA. Similarly, the common SA of team 
members depends on the correctness of its own (individual) SA. TSA (cognition and 
belief) also rests on its own correct cognition of the situation. When own cognition is 
correct, common correct cognition is high, leading to the high belief that other team 
members have also understood the situation in the same way. Moreover, common SA 
itself is also highly correlated with the TSA as a cognition and belief model. Therefore, 
the high level of common SA results in improved belief about the team member?s SA. 
Hence, all these TSA models seem interrelated and also dependent on the individual SA.  
Consequently, the design of systems supporting TSA should not only concentrate on 
the team communication and co-ordination but also on the individual SA since a major 
part of TSA depends on it. Moreover, instead of concentrating on one of these TSA 
218 
 
models, the system design should consider all three as they complement each other and 
improve the resulting team performance. If one of them is missing, TSA will be flawed.  
7.9 Implementation Requirements of SAVER  
The functioning and usefulness of SAVER has been demonstrated by the prototype but 
a considerable amount of work is needed to make it useable in a real environment. This 
section provides an overview for the development and deployment of the operational 
SAVER. It covers four main areas namely: organisational requirements, architectural 
requirements, user interface requirements and testing requirements to develop and 
deploy operational SAVER. The following sections discuss these requirements over a 
very broad spectrum.  
7.9.1 Organisational requirements  
An in depth requirements analysis is needed to develop and maintain situation 
awareness of individuals and teams in the organisation under focus. The process of 
requirements analysis should consider all the emergency management roles and their 
tasks depending upon the emergency scenarios. Moreover, multi-organisation co-
ordination requires agreement between organisations and comprehensive policies for 
information sharing and support.  
7.9.2 Architecture requirements 
The architecture of the SAVER system is discussed below in terms of various 
components. While the architectural specification is to some extent provisional and 
subject to revision based on the results of further research, it is included in the context 
of the current discussion as a means of indicating a proposed implementation strategy 
for ontology based information systems to increase the operational effectiveness of 
emergency managers in emergencies by improving their situation awareness. Figure 7.3 
(over page) shows a possible architecture of the operational SAVER.  
7.9.2.1 Knowledge repository 
The knowledge repository can consist of a LargeTripleStore1 to provide an adequate 
storage mechanism for the knowledge infrastructure of the application. With SAVER, 
                                                 
 
1 http://www.w3.org/wiki/LargeTripleStores 
219 
 
the knowledge infrastructure contains both conceptual knowledge and instance 
knowledge. The conceptual knowledge is represented using the concepts and 
relationships between the concepts, whereas the instance knowledge is in the form of 
individuals or values and the relationships between values and objects. 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
Figure 7.3 Architecture of operational SAVER  
7.9.2.2 Knowledge management 
The knowledge management module can make use of the tools to add, edit and append 
the ontologies in the system. These changes can come up due to the deficiencies in the 
existing models or may require updating in the dynamic emergency management 
 Knowledge management Query Management Visualisation 
Query 
Repository 
Triple Store 
Reasoning 
module 
Field Data 
Information 
Fusion 
SAVER 
Controller 
Storage 
Online 
resource
Query 
Engine 
220 
 
environment over the time. Third party tools like Prot?g?1, NeOn Toolkit2, or TopBraid 
Composer3 etc. can be used for ontology engineering and management.  
7.9.2.3 Reasoning module  
The reasoning module exploits any reasoner for asserting the knowledge into the 
reasoning environment and retrieving the results of inference execution. Moreover, the 
reasoning module is also used to check the ontologies for any inconsistencies in the 
relationships or rules. The reasoner used in the current application is Fact Plus Plus, 
which is simple to work with and efficient for small size ontologies, however, various 
other reasoners, for example, Racer, Pellet, Hermit etc. can be evaluated to choose the 
best one (Dentler et al., 2011; Huang et al., 2008 ).  
7.9.2.4 Query repository  
The query library consists of a collection of predefined query templates that may be of 
generic use across multiple operational contexts. Currently, SAVER uses only a few 
queries saved in SAVER Controller (next section) for different situations and roles. 
However, a wide range of queries can be saved using ontological characterisation to 
facilitate more meaningful and rapid selection and retrieval. In addition to this, the 
results of most generic queries can also be saved for quick retrieval.  
7.9.2.5 SAVER controller 
The controller component of SAVER is like the central processing unit of a computer. 
It can be activated by the feeds from the sources to which it is subscribed or the user 
can manually activate it by adding an event into SAVER and asking it to monitor some 
specific information sources. SAVER controller can then make use of available 
information, the reasoner module and predefined queries to provide the situation 
assessment to the users. The SAVER controller works as a communication channel 
between the users and the SAVER system. All messages, notifications and event 
management is done by the controller. 
                                                 
 
1 http://protege.stanford.edu/ 
2 http://neon-toolkit.org/wiki/Main_Page 
3 http://www.topquadrant.com/products/TB_Composer.html 
221 
 
7.9.2.6 Information fusion module  
An information fusion module can be used to integrate the information from diverse 
sources including information from within the SAVER system, for example from 
ontologies (TripleStore). The information is saved in the form of instances of ontologies 
and processed using a reasoning module to draw inferences according to the situation 
requirements.  
7.9.2.7 Computing and hosting requirements 
For a system like SAVER, quality and availability of service are very critical for the 
successful usage and trust of the emergency management sector. Both of these 
mandates come with huge cost and dedicated network resources. However, this cost can 
be reduced by using computing and storage facilities provided by cloud computing. In 
cloud computing there is a significant workload shift. Local computers no longer have 
to do all the heavy processing when it comes to running the huge applications (SAVER 
using large ontologies will require a huge amount of computing power). Moreover, using 
cloud computing, the hardware and software demands on the user?s side also decrease 
(Strickland, 2008). The only thing the user?s computer needs to be able to run is the 
cloud computing system?s interface software, which can be as simple as a web browser.  
SAVER is a web based application; therefore it can be divided into two sections - front 
end and back end. They connect though a network, usually the Internet. The front end 
is the side the computer users see; therefore, it can have very ordinary processing 
capability and only requires a web browser. The back end is the cloud section of the 
system, which is stored on the cloud in a secured environment and the cloud service 
providers assure its security. All the processing is done on the server side and hence the 
client side does not require the specialised hardware and software for running and 
storing large applications (see Figure 7.4). The reliability (and hopefully the security) of 
SAVER services will be assured by the cloud service providers. SAVER will become 
scalable in terms of storage space as well as the power it requires for processing the 
large and complex ontologies. Therefore, it can easily be extended to many other types 
of disaster contexts, organisations and thousands of users over the Internet.  
 
 
222 
 
 
 
 
 
 
 
 
 
 
Figure 7.4 SAVER on cloud 
7.9.3 Graphical user interface (GUI) requirements 
The design and implementation of the GUI components need to take into account the 
specific visualisation requirements of different user groups. Moreover, the ergonomic 
configuration of such interfaces must consider the cognitive requirements of the end 
users. This is of significant importance for providing the interfaces to improve the 
situation awareness of individuals and teams. For this purpose detailed user interface 
requirements analysis is needed. This analysis should involve emergency managers in the 
design process and assure the availability of information in the format required by them.  
7.9.3.1 SA, SSA and TSA oriented visualisation 
Interfaces should be designed by considering the factors that affect the situation 
awareness. For this purpose, as suggested in the design of the SAVER prototype, 
information should be arranged according to the goals and tasks of the roles. Moreover, 
the format of the information, i.e. pictorial, map, or text, should be used according to 
the requirements of the task. Adequate interfaces for chat and co-ordination should be 
provided along with the shared interface providing the overall up-to-date picture of the 
situation to improve the SSA and TSA of the team. Hence, all the design guidelines 
discussed in Chapter 5 and Chapter 6 should be used to improve SA, SSA and TSA of 
the individuals and teams by providing personalised interfaces for each role.  
SAVER Server 
Internet 
223 
 
7.9.3.2 Query visualisation  
The process of specifying appropriate queries is vital in the exploitation of the 
knowledge based systems. The queries central to the requirements of situation 
awareness need to be stored in the query repository as discussed above. Appropriate 
interfaces should be designed to allow building and execution of the customised queries. 
The interface should provide a way to build semantically enriched queries to take full 
advantage of an ontology based system. A very useful aspect in this regard can be the 
graphical query interface since, it does not require the users to know the knowledge 
architecture of the system. This area is still under research.  
7.9.3.3 Use of mobile devices and touch screens 
With the extensive improvement in the computing and graphical capabilities of mobile 
devices, they are now perfect candidates to keep the users up to date with the prevailing 
situation anywhere any time. SAVER is a web based application; therefore, it can be 
accessed on any device that is capable of running an Internet browser application. 
However, there is a need to develop adaptive interfaces so that a device can display 
information according to its capabilities. Moreover, large displays and multi-touch 
screens provide a new mode of visualisation and human computer interaction. Large 
screens can be used to provide a shared overall tactical picture of the prevailing situation 
improving team situation awareness. Similarly, touch screens can reduce the amount of 
learning required for information retrieval using textual queries or time consuming input 
by ordinary devices (e.g. keyboard, mouse, light pen) since maps and graphical 
components can be directly touched, dragged and dropped for input in a natural way. A 
combination of graphical query interfaces with the multi-touch screen would be an 
astonishing application of both these technologies.  
7.9.4 Testing requirements 
SAVER has been evaluated for improving the situation awareness of individuals and 
teams. However, before using the operational SAVER in real events, it needs to be 
tested with real data for accuracy, especially with large data sets. This can be done by 
using SAVER in multi-organisational exercises or in actual events as a dummy system. A 
detailed functional testing will verify that it provides the functionality to perform all the 
required tasks. Non-functional testing will clarify other aspects like availability, 
scalability, performance, security, etc.  
224 
 
7.10 Summary  
This research started with the requirement of an information system that can support the 
operations of managers in group emergency operation centres (GEOCs) during 
emergency situations such as mass evacuation. During the initial investigation of the 
literature and observations, it was noticed that emergency managers had mainly two types 
of responsibilities: 1) decision making, and 2) task performance. Decision making can be a 
requirement of a task or the task can be the result of the decision in the form of 
implementation. Both of these conditions require awareness of the situation. Since 
computer based systems are used in our everyday life as a solution to many problems, 
using them for improving SA seems entirely logical. Some questions linked to this idea 
were: how is situation awareness developed? What are its requirements? How to design a 
computer based system to support SA? Lastly, can a computer based system improve SA?  
These questions started the quest for the requirements of SA. Information about the 
situation appears to be the main requirement and so the study moved to the 
identification of these requirements. Literature, as well as common sense, indicate that 
the greater the awareness of a situation, the better the decisions made. The performance 
of tasks similarly also depends upon the degree of situation awareness. However, it is 
the relevance (quality), not the volume of information provided to develop SA, that is 
the key factor so care is needed to select only the information that needs to be 
processed for the decision or task at hand, thereby balancing workload and 
completeness. 
In emergency situations, acquiring situation awareness is more challenging than in 
normal situations due to stress levels, limited time, changing situation, etc. In this study, 
the information requirements are acquired by a Cognitive Task Analysis (CTA) based 
technique called Extended Goal Directed Information Analysis (EGDIA). This 
technique has the ability to capture the information requirements of cognitive processes 
along with the physical tasks. EGDIA is quite useful and comprehensive information 
requirements are gathered for various roles of GEOCs.  
This research focuses on emergency situations involving a mass evacuation following a 
tsunami threat. As a result of the EGDIA steps, the mass evacuation operation is 
divided into five phases, namely; pre-confirmation phase, tsunami-confirmed decision 
phase, evacuation phase, post-evacuation phase, return phase. Information requirements 
225 
 
are classified according to the roles and the phases of a mass evacuation operation, 
depending upon the tasks that need to be performed during each phase. These 
information requirements were validated via a questionnaire from emergency managers 
all over New Zealand. Very few additional comments were received and almost all the 
information requirements are declared relevant.  
Based on these information requirements, the next stage involved the design of a 
computer based system. While reviewing the literature for technology solutions to 
improve situation awareness, ontology based situation awareness involving domain 
concepts and relationships was seen to offer a promising approach to the design 
process. Ontologies of tsunami situations and information requirements were therefore 
developed, along with information of roles and their tasks in the different emergency 
phases previously identified. 
It is worth noting that existing ontology based SA systems only use the ontologies for 
modelling situations. However, the idea of the current research is to extend the existing 
ontology based SA to classify the situation information at Endsley?s three levels of SA 
using an ontology reasoner (Fact Plus Plus) for Level 3 predictions. The idea led to the 
design of a systems architecture and the implementation of a proof-of-concept 
prototype labelled Situation Aware Vigilant Emergency Reasoner (SAVER).  
The next phase employed SA-oriented design principles and a user centred design 
approach to develop a user interface for SAVER from which ontology based context 
reasoning provides personalised information to the various roles and so reduces 
information overload. SAVER was then evaluated in simulated experiments to see if it 
could improve the SA of the emergency managers. Situation Awareness Global 
Assessment Technique (SAGAT) was used to measure the SA of the participants 
obtained with and without SAVER. Results show that the SA of the SAVER users was 
significantly better than the non-SAVER users on all the SA levels. Moreover, the 
results illustrate that the SA of the less experienced emergency managers can also be 
improved to the level of the experienced emergency managers. Furthermore, the time 
taken by the SAVER users to retrieve SA for task use is also significantly shorter than 
the non-SAVER users, hence improving the efficiency of task performance.  
For large emergency management operations, team co-ordination is very important for 
successful performance of the collaborative tasks. To work in an harmonious way, the 
226 
 
team members need to share the information within the team in a timely manner. The 
ability to co-ordinate decisions and tasks depends upon the degree of shared situation 
awareness (SSA) that team members have about the related information requirements of 
their colleagues. SAVER supports SSA by proactively reminding every team member 
about the SA requirements of team members that depend upon him/her. The SSA of 
the participants was measured along with their individual SA measurement. Results 
indicate the SSA of the SAVER users is significantly superior to the non-SAVER users.  
Moreover, individuals in the team support their team members by successfully 
performing their own tasks or by contributing to other team member?s tasks. Therefore, 
by improving an individual?s SA, SAVER also improves this form of team SA (TSA). 
Another form of TSA, needed for the team decision making, is the common 
understanding of the situation. SAVER improves this understanding in three ways. 
Firstly, by providing the explicit situation assessment, secondly, by sharing a related 
team member?s SA, and thirdly, by improving the individual SA. One more form of 
TSA is a team member?s degree of awareness about the other members? situation 
awareness. This is required so that the other team members? SA can be updated or 
supported, if required. SAVER supports this form of TSA by improving the common 
understanding of a situation, by providing explicit situation assessment and by sharing 
the team member?s SA. All these forms of TSA are measured in simulated experiments 
with and without using SAVER. Results prove that the TSA of the SAVER users in all 
forms of TSA is significantly larger than the non-SAVER users, individually and as a 
team. Figure 7.5 summarizes the evolution of SAVER through the three chapters (4-6) 
of this thesis.  
 
 
 
 
 
 
 
227 
 
 
 
 
 
 
 
 
 
 
Figure 7.5 SAVER?s evolution to support SSA and TSA 
Briefly, EGDIA proves to be very useful in gathering comprehensive information 
requirements and the use of ontologies is helpful for designing computer systems 
supporting SA, SSA and TSA. Ontologies can be used for computer based automatic 
situation assessment and so such a system can be used for supporting human SA. 
Furthermore, SA-oriented computer system design using precise information 
requirements improves the SA, SSA and TSA of the emergency managers. Use of a 
computer based SA improves the effectiveness and efficiency of the individual and team 
decision making by allowing them to make informed decisions in a short period of time. 
Additionally, a system with contextualised information improves the team co-ordination 
and collaboration required in  large scale operations such as a mass evacuation, 
consequently improving the overall emergency management.  
Chapter 8 contains recommendations for future work that will explore the implications 
of developing SAVER for emergency management and applying the concept to other 
domains. 
 
 
 
SAVER 
Situation Assessment  
SA Level 1 
SA Level 2 
SA Level 3 
SAVER 
SA 
Situation 
Assessment User?s context 
Task 
requirements 
SAVER 
SSA-TSA 
Situation 
Assessment 
Team 
member?s SA 
Team members? 
Task 
requirements 
228 
 
 
SUMMARY, CONCLUSIONS 
AND FUTURE WORK 
?Each problem that I solved became a rule, which served afterwards to solve other problems? 
(Rene Descartes) 
 
This chapter provides a summary of the discoveries and other outcomes of the research 
work described in this thesis. It also includes the main conclusions associated with the 
points made. More detailed information on the conclusions is contained in the 
Discussion (Chapter 7).   
8.1 Research Questions Answered 
This first section demonstrates the scope and success of the research by reiterating the 
research questions and describing how they were answered. Cross references are given 
to the sections in the thesis where the relevant issues are discussed in full. 
8.1.1 How to design a computer based system for automatic situation assessment 
based on situation awareness (SA) requirements in emergency situations? 
Chapter 4 of the thesis explains the design of a computer based system (SAVER) to 
support emergency managers in mass evacuation operations. Information requirements 
of the emergency managers to develop and maintain SA are gathered for the various 
emergency management roles. Moreover, a method to classify the incoming information 
according to the requirements of the emergency managers and the levels of SA is 
devised.  
a. How is SA developed in emergency situations?  
An extensive review of the literature is carried out to understand SA development and 
its implications for emergency situations. Relevant literature is covered in Section 2.4 of 
Chapter 2, to describe the theoretical underpinning of the concept and various 
229 
 
processes involved in the SA development. However, the various systems supporting 
SA are discussed in Section 2.6.3 of Chapter 2.  
b. How to acquire the dynamic information needed for developing and maintaining 
situation awareness (SA)? 
Extended Goal Directed Information Analysis (EGDIA) is used to gather the 
information requirements of developing and maintaining SA. It is a cognitive task 
analysis (CTA) technique since the information requirements for the cognitive processes 
are also captured along with the physical task. This method was quite successful in 
gathering the comprehensive information requirements. Details of this technique and 
the resulting information requirements are described in Chapter 4, Section 4.4.1.  
c. How to classify the information requirements according to the SA levels and the 
phases of an emergency situation? 
Firstly, a mass evacuation operation is divided into five different phases covered in 
Section 4.4.1.2 of Chapter 4. Secondly, the information requirements are classified 
according to the roles and phases of a mass evacuation operation. Moreover, the 
information requirements are also classified according to the three levels of SA 
(perception, comprehension and projection) to determine what information is required 
to develop each level of SA. For automatic classification of the SA information, 
ontologies of a tsunami situation (domain ontology) and the information requirements 
are developed. An ontology reasoner (Fact plus plus) is used for the validation and 
classification of the ontologies. The architecture design of an ontology based SA is 
covered in section 4.5 of Chapter 4.  
d. Whether ontologies can be used to classify SA information or not? 
The study demonstrated that ontologies can be used to classify the SA information. 
Since they can model the situations in great detail, the use of rules and axioms allows the 
inference required for the classification of elements as class types. Moreover, the 
classification of SA information was tested for the simulated scenarios. Ontology based 
reasoning successfully provided the automatic situation assessment according to the SA 
levels. Details of the evaluation are in section 4.5 of Chapter 4.  
230 
 
8.1.2 How to design a computer based system supporting human SA in mass 
evacuation operations? 
Chapter 5 of the thesis explains the user interface design of the computer based system 
(SAVER) to support human SA in mass evacuation operations. On the one hand, user 
interfaces are designed to provide the support to human SA, and on the other, the same 
interfaces are used to see if computer based systems can improve the SA of individuals 
in emergency situations. The SA of the emergency managers is improved on all three 
levels by providing them with the information needed to develop the SA necessary for 
decision making and task completion according to their roles.  
a. How to design a user interface for a computer based system supporting SA? 
The user interface of the computer based SAVER system is designed using the SA 
oriented design principles. The most important of these design principles is about 
delivering the processed information directly for developing SA Level 2 and Level 3 
awareness. Details of these design principles and user interfaces are provided in Chapter 
5, sections 5.4.3 and 5.4.4 respectively.  
b. How to reduce the information overload of emergency managers? 
The information overload of emergency managers can be minimised by providing the 
personalised information according to the roles and tasks they perform. For this 
purpose, the context information regarding the emergency managers, their roles, 
responsibilities and tasks are modelled in ontologies. These ontologies, along with the 
ontologies developed in Chapter 4 for tsunami situations (domain ontology) and the 
information requirements, are used to provide the personalised information to the 
emergency managers. Ontology based context design is explained in section 5.4.2 of 
Chapter 5.  
c. Whether a computer based system can improve SA of emergency managers? 
The study demonstrated that a computer based system can improve the SA of 
emergency managers in emergency situations. This is confirmed by measuring the SA of 
emergency managers using a technique called Situation Awareness Global Assessment 
Technique (SAGAT). The SA of emergency managers was measured using, and without 
using, the computer based system and the scores were compared. The reason for 
231 
 
measuring SA was twofold, first to check whether SA can be improved using computer 
based systems, and secondly, to evaluate the design of the computer based system 
developed for SA support. After the evaluation, it is confirmed that the developed 
system is very useful in improving the SA of emergency managers. Details of the SA 
measure and the evaluation of the computer based system are reported in the section 5.5 
of Chapter 5 and discussed in section 7.5 of Chapter 7.  
d. How to evaluate the user interface design developed for computer based system 
supporting SA? 
Apart from the evaluation by measuring individual SA, user interfaces are also evaluated 
for the human computer interaction (HCI) parameters like usability, e.g. ease of use, 
understandability, learnability, functionality, etc. All interfaces were positively evaluated 
by the emergency managers and very few modifications were suggested. Details of the 
HCI evaluations are reported in section 5.5.3 of Chapter 5.  
8.1.3 How to design a computer based system supporting shared and team SA 
(SSA and TSA) of the emergency managers in mass evacuation 
operations? 
Chapter 6 describes the design, development, and evaluation of a computer based 
system to support the SSA and TSA of emergency managers in mass evacuation 
operations.  
a. How are SSA and TSA developed for decision making and collaborative tasks? 
SSA is improved by reminding every team member about the requirements of other 
team members who are dependent upon them. Similarly, TSA is developed by sharing 
the team members? understanding of the situation with other relevant team members. 
Moreover, TSA as a common understanding of the situation, is also developed by 
providing the explicit situation assessment with logical arguments and improving the 
individual SA. Lastly, TSA as a belief about the team members? SA, is developed by 
improving the common understanding of the situation. Details of SSA and TSA 
development are covered in section 2.5 of Chapter 2, section 6.4.1 of Chapter 6 and 
sections7.6 and 7.7 of Chapter 7.  
 
232 
 
b. How can ontology based contextual information be used to improve SSA, and TSA? 
Contextual information is used to identify which team members are working on the same 
goals and tasks. Moreover, it can be used to understand which information is required by 
various roles for different tasks. Hence, the source and destination of the information can 
be used to improve their SSA by reminding the source to supply information manually to 
the destination thereby improving the recipien?ts SA. Similarly, TSA can be improved by 
automatically identifying the common goals and tasks, that will answer ?whose SA needs 
to be shared with whom?? Like automatic situation assessment, automatic context 
reasoning is done using ontology based reasoning. Ontology based SSA and TSA oriented 
design is presented in section 6.4.2 of Chapter 6.  
c. Whether a computer based system can improve the SSA and TSA of emergency 
managers? 
A computer based system can significantly improve the SSA and TSA of emergency 
managers in emergency situations. This result is established by measuring the SSA and 
TSA of managers in the simulated experiments using, and without using, a computer 
based system. The measured SSA and TSA are significantly better when SAVER is used. 
Details of SSA and TSA measurement and the evaluation of the computer based system 
are reported in section 6.5 of Chapter 6 and discussed in sections 7.6 and 7.7 of Chapter 7.  
8.2 Future Work 
Due to the myriad applications where SA, SSA and TSA are important, this study has 
widespread utility and potentially enormous impact. To conclude, we illustrate some of 
the key fields that can make use of research results with little extension.  
8.2.1 Option Awareness 
In emergency situations, once the required level of SA, SSA and TSA is achieved the 
decision makers have to plan an action to normalise the situation. The action(s) comes 
from the different choices known to the decision makers. Clearly, knowledge, or lack of 
it, of the options, can affect the quality of the decisions made. This option knowledge is 
called option awareness (Pfaff et al., 2010). Hence, like situation awareness, option 
awareness can also be improved by extending SAVER to analyse various options 
according to the prevailing situation. The architecture of such systems can use automatic 
situation assessment by SAVER to improve option awareness by presenting options to 
233 
 
managers and recommending (but not deciding) preferred alternatives with reasons and 
consequences. A complete package for effective and efficient decision making could be 
developed.  
Related to option awareness, further research is needed to understand the impact of 
computer based systems on the quality of decisions and the tasks performed. This 
would help to develop an in-depth understanding of the relationship between the degree 
of SA and the variation in quality of decisions and outcomes. Moreover, option 
awareness can provide a link between the degree of situation awareness and the quality 
of decision made.  
8.2.2 Agents and Trust 
With the advent of computer based SA systems, a framework is needed for defining the 
SA of computer agents. This will help to identify the attributes of computer agents who 
possess SA capability and the capability to improve it. More work is needed to study the 
extent to which computer based SA can be used and trusted in real life scenarios. It was 
noticed that in the simulated experiments emergency managers seemed to accept the 
suggestions of computer based systems but in actual events the situation may be very 
different. Hence, research is needed to identify those attributes that can improve trust. 
Work is even needed to find the extent to which human decision makers trust computer 
based systems.  
A more recent theme to emerge in the SA literature is the concept of distributed or 
systemic SA. Distributed SA (DSA) approaches are born out of distributed cognition 
theory (Hutchins, 1995), which describes the notion of joint cognitive systems 
comprising the people in the system and the artefacts they use. Within such systems, 
cognition is achieved though co-ordination between the system units (Artman & Gabris, 
1998). DSA approaches therefore, view TSA not as a shared understanding of the 
situation, but rather as an entity that is separate from team members and is in fact, a 
characteristic of the whole system. The SA of a team is distributed not only throughout 
the agents comprising them, but also in the artefacts they use in order to accomplish 
their goals. Therefore, it would be useful to consider SAVER as part of an overall 
system and evaluate the interaction of various human and computer agents within the 
system.  
234 
 
The efficiency of the ontology based systems is also a question since even the latest 
ontology reasoners have problems in classifying large and complex ontologies. More 
research is needed to improve the efficiency of ontology based reasoning. Moreover, 
standard methods are needed for the development and maintenance of large and 
complex ontologies.  
8.2.3 Integration of various disaster management sectors; health, fire, police  
The current architecture of SAVER is designed in a way that it can be extended to other 
disaster management sectors like health, police, fire service etc., to provide one system for 
all. Since in most large scale operations these organizations work together, an extensive 
requirements analysis can be done to identify their information and organisational needs. 
These requirements can be analysed to reveal the shared and team SA requirements. 
Having a single integrated system has many advantages such as standardised 
communication between organisations/departments and the development of an overall 
situation picture that will also help to improve SSA and TSA.  
Rules and axioms of ontologies can be used to define which information to share and 
when. In this way, an overall integrated system composed of small individual systems 
can be developed. Such a system can fulfil local requirements and at the same time allow 
inter-operability on a regional, national or global scale. Moreover, the information 
integration from diverse sources, including web sources, will provide a better overall 
picture to the decision makers by acquiring the information from any possible valid 
source. Emergency management plans and guides can also be made part of such systems 
for automatic planning support.  
8.2.4 Training of emergency managers 
A system like SAVER can be very useful for the training of emergency managers. It can 
be used to create various simulated scenarios and to generate information flows between 
various roles. Emergency managers can be trained for various possible situations. In 
addition, their performance can be measured by measuring their SA and improvement 
in their performance can be monitored over time. By changing different factors, e.g. 
information presentation styles, stress levels, available time etc., various behavioural 
studies can be performed to disclose different problems of emergency decision making. 
Various solutions for complex problems can also be tested or evaluated, as this study 
does.  
235 
 
8.2.5 Extension to mobile technologies 
In recent years, with the rapid growth of wireless and mobile technologies, users can 
make use of advanced technologies on the go, with no time or location constraints 
(Wang et al., 2009). This effective connection allows them to remain aware of the 
situation all the time. Moreover, if they are in the field, they can use various features of 
mobile devices, e.g. cameras, to add pictorial and audio information to the computer 
based system that can be shared to improve the situation awareness of the whole team 
of decision makers. Lai et al., (2007) studied the affordance of mobile technologies 
compared to desktop computers for real time information capture (wherever and 
whenever) and rapid transmission to corporate systems. They found the mobile aspect 
of considerable value and its application to emergencies (provided a broadband service 
is still available) is very clear. The extension of SAVER to mobile technologies can be 
done by expanding the context ontologies of SAVER with the features of mobile 
devices and adding the use methods for the integration of information from them.  
8.2.6 Business decision making 
Many emergency decision making theories came from the background of business and 
defence (military) decision making. This is not a one way transfer. The digitisation and 
computerisation of emergency management tools can now be directed to business 
decision making and even to the re-engineering of business processes. Business decisions 
can be enhanced by improving the situation awareness of the decision makers. In 
addition, business decision making frequently emphasises costs, benefits and risks and 
modelling business decisions with ontologies can provide different and more complete 
perspectives leading to greater efficiency, better collaboration and competitive advantage. 
8.2.7 Military applications 
Like emergencies, the decision making environment in battlefields is very complex and 
dynamic since many different factors are in action. War gaming and emergency training 
scenarios have much in common including the need for rapid decision making in 
dynamic situations that demand high levels of situation awareness. Ontology based 
reasoning can be applied to those situations when it may be difficult to identify the 
factors (or options) of relevance to a situation. Similarly, in such scenarios, it may also 
be unclear how the information relevant to a particular factor should be analysed, 
operationalised and characterised making it difficult to interpret the significance of 
information in terms of its impacts on mission objectives. An ontology based design 
236 
 
such as SAVER could be of considerable value in the military domain to improve 
situation awareness with consequent benefits for personnel deployment, resource 
preservation, concerted action and the execution of strategy. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
237 
 
References 
Abowd, G. D., Dey, A. K., Orr, R., & Brotherton, J. A. (1997). Context-awareness in 
wearable and ubiquitous computing. In: Proceedings of the 1st IEEE International 
Symposium on wearable Computers, Washington, DC, USA. pp. 179?180. 
Adams, M.J., Tenney, Y.J., & Pew, R.W. (1995). Situation awareness and the cognitive 
management of complex systems. Human Factors, 37(1), 85-104.  
Adams, N.M., Field, M., Gelenbe, E., Hand, D.J., Jennings, N.R., Leslie, D.S., 
Nicholson, D., Ramchurn, S.D., Roberts, S.J., & Rogers, A. (2008). The 
ALADDIN Project: intelligent agents for disaster management. In: Proceedings of the 
EURON/IARP International Workshop on Robotics for Risky Interventions 
and Surveillance of the Environment. Benicassim, Spain, 7-8 Jan, 2008[CD-
ROM]. 
AEMI ? Australian Emergency      Management     Institute (2008). Emergency 
management. Available at: 
http://www.ema.gov.au/www/emaweb/emaweb.nsf/Page/Emergency_Manag
ement. Last accessed 15 March 2011.Albers, 1998 
Albers, M. J. (1998). Goal driven task analysis: improving situation awareness for complex problem 
solving. In: Proceedings of 16th Annual International Conference in Computer 
Documentation. Quebec, Canada, 1998. pp. 234-242.  
Albers, M.J. (2004). Communication of complex information: user goals and information needs for 
dynamic web information. Manwah NJ: Lawrence Erlbaum Associates Inc.  
Albert, D., & Steinberg, L. (2011). Judgement and decision making in adolescence. 
Journal of Research on Adolescence. 21(1). 211-223. DOI:10.1111/j.1532-
7795.2010.00724.x 
Albertsten, T., & Blomqvist, E. (2007). Describing ontology applications. European Semantic 
Web Conference (ESWC), (pp. 549-563). Innsburck, Austria.Amalberti and 
Beblon.  
Ankolekar, A., Burstein, M., Hobbs, J. R., Lassila, O., Martin, D., McDermott, D., 
McIlraith, S. A., Narayanan, S., Paolucci, M., Payne, T., & Sycara, K. (2002) 
238 
 
AML-S: Web service description for the semantic web. In: Semantic Web - ISWC2002 
(Vol. 2342, pp. 348-363). Berlin: Springer-Verlag Berlin. 
Artman, H. (2000). Team situation assessment and information distribution. Ergonomics, 
43(8), 1111?1128. 
Asthana, A., Cravatts, M., & Krzyzanowski, P. (1994). An indoor wireless system for 
personalized shopping assistance. In IEEE Workshop on Mobile Computing Systems 
and Applications, Santa Cruz, CA, US.  
Aziz, Z., Pen~a-Mora, F., & Chen, A., (2009). Supporting urban emergency response 
and recovery using RFID-based building assessment. Disaster Prevention and 
Management, 18(1), 35-48.  
Baclawski, K., Kokar, M., & Matheus, C. (2003) Formalization of situation awareness. In H. 
Kilov (Ed.), Behavioral Specifications of Businesses and Systems. London: 
Kluwer Academic Publishers. 
Bailey, B. P., Konstan, J. A., & Carlis, J. V. (2000). Measuring the effects of interruptions ontask 
performance in the user interface, IEEE International Conference on Systems, Man, 
and Cybernetics, Vol. 2, pp. 757?762, 2000.  
Baumgartner, N.,  Retschitzegger, W., & Schwinger, W., (2008).  Application scenarios for 
ontology-driven situation awareness systems - Exemplified for the road traffic management 
domain, Proceedings of the 3rd Workshop on Formal Ontologies Meet Industry 
(FOMI), Torino, Italy, IOS Press, 2008, pp. 77-87 
Baumgartner, N., Gottesheim, W., Mitsch, S., Retschitzegger, W., & Schwinger, W. 
(2010). Editorial: BeAware!-Situation awareness, the ontology-driven way. Data 
Knowl. Eng., 69(11), 1181-1193. 
Beirne, M., Ramsay, H., & Panteli, A., (1998). Participating informally: Opportunities 
and dilemmas in user driven design. Behaviour and Information Technology, 17(7), 
301-310.  
Bellardo, S., Karwan, K.R., & Wallace, W.A., (1984). Managing the response to disasters 
using microcomputers. Interfaces, 14(2), 29-39.  
 
239 
 
Benerecetti, M., Bouquet, P., & Bonifacio, M. (2001). Distributed context-aware systems. 
Human Computer Interaction. Special issue on Context-aware computing, 16:213?228, 
2001. 
Bergstrand, F., & Landgren, J. (2009). Using live video for information sharing in 
emergency response work. International Journal of Emergency Management. vol 
6(3/4). Indersciences Enterprises Ltd.  
Berkum, M. M. (1964). Performance decrement under psychological stress. Human 
Factors 6, 21?30. 
Berners-Lee, T., Hendler, J., & Lassila, O. (2001) The Semantic Web. Scientific American, 
284(4), 34-43. 
Betts, B. J. (2005). Improving situational awareness for first responders via mobile computing. 
Moffett Field, Calif, National Aeronautics and Space Administration, Ames 
Research Center. http://purl.access.gpo.gov/GPO/LPS123514. 
Bevan, N. (2001). International standards for HCI and usability. 
International Journal of Human-Computer Studies, 55(4), 533-552 
Beyer, H., & Holtzblatt, K., (1998). Contextual design: Defining customer centred systems. San 
Francisco, CA: Morgan Kaufmann Publishers.  
Billings, C. E. (1995). Situation awareness measurement and analysis: A commentary. 
Proceedings of the International Conference on Experimental Analysis and 
Measurement of Situation Awareness (Daytona Beach, FL: Embry-Riddle 
Aeronautical University Press). 
Blachowicz, J. (2009). How science textbooks treat scientific method: A philosopher's 
perspective. British Journal for the Philosophy of Science , 60(2), 303-344. 
Blandford, A., & Wong, W. B. L. (2004). Situation awareness in emergency medical 
dispatch. International Journal of Human Computer Studies, 61 (4), pp. 421-452.  
Bolstad, C. A., & Endsley, M. R. (2000).The effect of task load and shared displays on team 
situation awareness. In Proceedings of the 14th Triennial Congress of the 
International Ergonomics Association and the 44th Annual Meeting of the 
Human Factors and Ergonomics Society. Santa Monica, CA: HFES 
240 
 
Bolstad, C. A., & Endsley, M. R. (2003). Measuring shared and team situation awareness in 
army?s future objective force. Proceedings of the Human Facotrs and Ergonomics 
Society Annual Meeting 2003 47: 369.  
Bolstad, C. A., Cuevas, H. M., & Costello, A. M. (2005). Improving situation awareness 
through cross training. Human Factors and Ergonmics Society 49th Meeting, 
October 2005, Orlando, FL.  
Bolstad, C. A., Castello, A. M., & Endsley, M.R., (2006). Bad situation awareness designs. 
What went wrong and why. In Proceedings of 16th Congress of International 
Ergonomics Association. Maastricht, Netherlands, 10-14 July 2006.  
Bond, S., & Cooper, S. (2005). Modelling emergency decisions: recognition primed 
decision making. The literature in relation to an ophthalmic critical incident. 
Journal of Clinical Nursing, 15, 1023-1032 
Boury-Brisset, A. C. (2003) Ontology-based approach for information fusion. Paper presented at 
the 6th International Conference on Information Fusion, Cairns,Queensland, 
Australia. 
Boyd J. (1987). A discourse on winning and losing. In Unpublished briefing slides: Air 
University Library, Maxwell AFB. 
Brooks, F., (1987) No silver bullets: essense and accidents of software engineering, 
IEEE Computer 20 (4), 10-19.  
Bryant, D.J. (2002). Making naturalistic decision making ?Fast and Frugal?. In: Proceedings of 
the 7th International Command and Control Research and Technology 
Symposium. Quebec City, ON, Canada, September 2002. Available at: 
http://pubs.drdc.gc.ca/PDFS/unc48/p523444.pdf.  Accessed on April 2011. 
Bunge, M. (1984). Philosophical inputs and outputs of technology, in G. Bugliarello and 
D. Donner (eds.) History and Philosophy of Technology, University of Illinois Press, 
Urban, IL, 263-281.  
Burke, E., Hendry, C. (1997), "Decision making on the London incident ground: an 
exploratory study", Journal of Managerial Psychology, Vol. 12 pp.40-7. 
241 
 
Carver, L., & Turoff, M. (2007). Human computer interaction: the human and computer 
as a team in emergency management information systems. Communications of the 
ACM, 50(3), 33-38.  
Checkland, P.B . (1999). Soft systems methodology in action, John Wiley and Sons Ltd, 
Chichester, ISBN: 0471986054.  
Chen, H., Finnin, T., & Joshi, A. (2004). An ontology for context-aware pervasive 
computing environments. Special issue on ontologies for distributed systems, Knowledge 
engineering reviews, 18 (3): 197 ? 207.  
Christopoulou, E., & Kameas, A. (2005). GAS ontology: An ontology for collaboration 
among ubiquitous computing devices. International Journal of Human Computer 
Studies, 62, 5, 664-685 
Churchman, C. W. (1971), The design of inquiring systems: Basic concepts of systems and 
organizations, Basic Books, New York, NY. 
Clark, R. E., Feldon, D., van Merrienboer, J. J., Yates, K., & Early, S. (2008). Cognitive 
task analysis.? In Handbook of Research on Educational Communications and 
Technology, 3rd ed., edited by J. M. Spector, M. D. Merrill, J. J. G.van 
Merrienboer, and M. P. Driscoll, 577?94. Mahwah, NJ: Lawrence Erlbaum 
Associates.  
Coen, M. H. (1998). Design principles for intelligent environments. In Proceedings of 
AAAI/IAAI 1998, pp 547?554, 1998. 
Cole, R., Purao, S., Rossi, M., & Sein, M. K. (2005). Being proactive: Where action research 
meets design research, In Proceedings of the 26th International Conference on 
Information Systems, Las Vegas, pp. 325-336. 
Collier, S. G., & Folleso, K. (1995). SARCI: A measurement of situation awareness for 
nuclear power plant control rooms. In D. J. Garland and M. R. Endsley (Eds.), 
Experimental analysis and measurement of situation awareness. Daytona Beach, F.L: 
Embry-Riddle University Press. 215-122.  
242 
 
Cort?s, U., S?nchez-Marr?, M., Ceccaroni, L., R-Roda, I., & Poch, M. (2000) Artificial 
intelligence and environmental decision support systems, Applied Intelligence, 13: 
77 ? 91.   
Courtney, J. F. (2001) Decision making and knowledge management in inquiring 
organizations: toward a new decision- making paradigm for DSS. Decision Support 
Syst. 31, 17-38 (2001). 
Cowan, N. (1988). Evolving conceptions of memory storage, selective attention, and 
their mutual constraints within the human information processing system. 
Psychological Bulletin, 104(2), 163-191. 
Cronan, K. (1998). Foundation of emergency management. Australian Journal of Emergency 
Management, 13(1), 20-23.  
Dardenne, A., Fickas, S., & van Lamsweerde, A. (1991), Goal-directed concept acquisition in 
requirements elicitation, Proceedings of 6th International Workshop on Software 
Specification and Design, pp. 14-21. 
Dastani, M., Hulstijn, J., & Van der Torre, L. (2003) How to decide what to do. European 
Journal of Operations Research. 
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of 
information technology. MIS Quarterly, 13(3), 319-340. 
De-Leoni, M., Mecella, M., de Rossa, F., Marrella, A., Poggi, A., Krek, A., & Manti, F. 
(2007). Emergency management: From user requirements to a flexible P2P architecture. In: 
Proceedings of the 4th International ISCRAM conference. Delft, Netherlands, 
13-16 May, 2007, pp. 271-279.  
Dentler, K., Cornet, R., Teije, A. T. and de-Keizer, N. (2011). Comparison of 
reasonersfor large ontologies in the OWL 2 EL Profile. IOS Press Journal: 
Semantic Web  Interoperability, Usability, Applicability 1(2011)(1-5).  
Dey, A. K. (2000). Providing architectural support for building context-aware applications. 
Doctoral dessertation, Georgia Institute of Technology, 2000. Retreived from:                                          
http://swiki-lifia.info.unlp.edu.ar/ContextAware/uploads/10/dey-thesis.pdf   
 
243 
 
Dominguez, C. (1994). Can SA be defined? In: Situation Awareness: papers and annotated 
bibliography, M. Vidulich, C. Dominguez, E. Vogel and G. McMillan (Eds) 
Report AL/CF-TR-1994?0085 (Wright-Patterson Airforce Base, OH: Air Force 
Systems Command). 
Downs J.S., & Fischhoff, B. (2009). Theories and models of adolescent decision making. In R.J. 
DiClemente, J.S. Santelli & R.A. Crosby (Eds.) Adolescent Health: 
Understanding and Preventing Risk Behaviors, San Francisco, CA: Jossey-Bass. 
Driskell, J. E., & Salas, E. (1991). Group decision making under stress. Journal of Applied 
Psychology 76, 473?478. 
Dreyfus, S.E. (1981). Formal models vs. Human situational understanding: Inherent limitations on 
the modelling of business expertise (ORC 81-3). Berkley: Operations Research Centre, 
University of California.  
Durso, F. T., Gronlund, S. D. (1999). Situation awareness. In: F. T. Durso, R. Nickerson, 
R. Schvaneveldt, S. Lindsay & M. Chi, eds. Handbook of Applied Cognition. 
New York: Wiley, pp 284-314.  
Durso, F. T., & Sethumadhavan, A. (2008). Situation awareness: Understanding dynamic 
environments. Human Factors, 50(3), 442-448.  
Elstein, A. S. (2001). Naturalistic decision making and clinical judgement. Journal of 
Behavioural Decision Making 14, 353?384. 
Endsley, M. R. (1988a). Design and evaluation for the situation awareness enhancement. 
In proceedings of Human Factors Society 32nd Annual Meeting Volume-1, pp 
97-101.  
Endsley, M. R. (1988b). Situation awareness global assessment technique (SAGAT). In 
Proceedings of National Aerospace and Electronics Conference (NAECON) 
IEEE, New York, pp 789-795.  
Endsley, M. R. (1989) Final Report: Situation awareness in an advanced strategic mission 
(NORDOC 89?32) (Hawthorne, CA: Northrop Corporation). 
244 
 
Endsley, M. R. (1990). Situation awareness in dynamic human decision making: Theory and 
measurement. Unpublished doctoral dessertation, Universoty of Southern 
California, Los Angeles, CA.  
Endsley, M.R. (1995a). A taxonomy of situation awareness errors. Human Factors in 
Aviation Operations. (287-292). Aldershot, England: Avebury Aviation, Ashgate 
publishing Ltd.  
Endsley, M.R. (1995b). Toward a theory of situation awareness. Human Factors, 37(1), 
32-64. 
Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical review, In: M. 
R. Endsley & D. J. Garland, eds. Situation Awareness Analysis and 
Measurement. Mahwah, NJL Lawrence Erlbaum Associates Inc., pp. 3-32.  
Endsley, M.R., (2001). Designing for situation awareness in complex systems. In: Proceedings of 
the Second International Workshop onSymboisis of Humans, Artefacts and 
Environment. Kyoto, Japan, pp. 176-190.  
Endsley, M .R., & Bolstad, C. A., (1993). Human capabilities and limitations in situation 
awareness. In: Combat Automation for Airborne Weapon Systems: Man / 
Machine Interface Trends and Technologies (AGARD-CP-520). Neuilly Sur 
Seine, France: NATO ? AGARD, pp. 19/1 ? 19/10.  
Endsley, M .R., & Bolstad, C. A. (1994). Indivdual differences in pilot situation 
awareness. International Journal of Aviation Psychology, 4(3), 241-264.  
Endsley, M. R., & Garland, D. J. (2000). Eds. Situation awareness analysis and 
measurement. Mahwah, NJ: Lawrence Erlbaum Associates Inc.  
Endsley, M.R., & Jones, W.M., (1997). Situation awareness information dominance & 
information warfare. Logicon Technical Services Inc. Dayton ? OH. Technical 
report 97-01. 
http://www.satechnologies.com/Papers/pdf/IW%26SAreport%20.pdf. 
Accessed March 2011.  
Endsley,   M. R., & Jones,  W.M.,   (2001),   A   model   of   inter-   and   intra-team   
situation awareness:   Implications   for   design,   training   and   measurement.   
245 
 
In   New   Trends   in Cooperative   Activities:   Understanding   System   
Dynamics   in   Complex   Environments, M. McNeese, E. Salas and M. Endsley 
(Eds) (Santa Monica, CA: Human Factors and Ergonomics Society). 
Endsley, M. R., & Robertson, M. M. (2000), Situation awareness in aircraft maintenance 
teams. International Journal of Industrial Ergonomics, 26, 301?25. 
Endsley, M. R., & Rodgers, M. (1994). Situation awareness information requirements for en route 
air traffic control. Washington, D.C: U.S. Dept. of Transportation, Federal 
Aviation Administration, Office of Aviation Medicine. 
Endsley, M. R., & Rodgers, M. D. (1998). Distribution of attention, situation awareness, 
and workload in a passive air traffic control task: Implications of operational 
errors and automation. Air Traffic Control Quaterly, 6(1), 21 ? 44.  
Endsley, M. R., & Smith, R. P. (1996). Attention distribution and decision making in 
tactical air combat. Human Factors 38(2), 232-249. 
Endsley, M. R., Bolstad, C.A., Jones, D.G., & Riley, J.M., (2003a). Situation awareness 
oriented design: From user?s cognitive requirements to creating effective supporting technologies. 
In: Proceedings of the Human Factors and Ergonomics Society 47th Annual 
Meeting. Denver, Colorado, USA, 13-17 October, pp. 268-272.  
Endsley, M. R., Bolte, B., & Jones, D.G. (2003b). Designing for situation awareness: An 
Approach to Human Centred Design. London: Taylor & Francis.  
Endsley, M. R., Farley, T. C., Jones, W. M., Midkiff, A. H., & Hansman, R. J. (1998). 
Situation awareness information requirements for commercial airline pilots. (ICAT-98-1). 
Cambridge, MA: Massachusetts Institute of Technology International Centre for 
Air Transportation.  
Endsley, M. R., Hoffman, R., Kaber, D., & Roth, E. (2007). Cognitive engineering and 
decision making: An overview and future Course, Journal of Cognitive Engineering 
and Decision making, 1(1), 1-21.  
Ericsson, K.A., & Kintsch, W. (1995). Long terms working memory. Psychological review, 
102, 211-245 
246 
 
FEMA - Federal Emergency Management Authority, (2007). Principles of emergency 
management supplement. FEMA?s higher education project, 2007. Available at: 
http://www.iaem.com/publications/documents/PrinciplesofEmergencyManag
ement.pdf. Accessed on 13 April, 2011.  
Ferreira, F., Dantas, A.,Seville, E., & Giovinazzi, S. (2009). Extreme events decision 
making in transport networks: A holistic approach using emergency scenarios 
and decision making theory. Journal of the Eastern Asia Society for Transportation 
Studies, ONLINE ISSN: 1881-1124. 
Fiedler, F. E. & Garcia, J.E., (1987). New approaches to leadership: Cognitive resources and 
organizational performance. Wiley, New York. 
Fikes, R., Hayes, P., & Horrocks, I. (2003). OWL-QL - A language for deductive query 
answering on the semantic web. Stanford, California: Knowledge Systems Laboratory, 
Stanford University. 
Fiore, S.M., Salas, E., Cuevas, H.M., & Bowers, C.A. (2003). Distributed coordination 
space: toward  a  theory  of  distributed  team  process  and  performance.  
Theoretical  Issues  in Ergonomics Science, 4, pp. 340?364. 
Flanagan, J. C. (1954). The critical incident technique, Psychological Bulletin, 54(4). Available 
at: http://www.apa.org/pubs/databases/psycinfo/cit-article.pdf 
Flin, R. (1996). Sitting in the hot seat: Leaders and teams for critical incidents. Chichester: John 
Wiley and Son.   
Flin, R., Staven, G., & Stewart, K. (1996). Emergency decision making in the offshore 
oil and gas industry. Human Factors 38, 262?277. 
Folmer, E., & Bosch, J. (2004). Architecting for usability: A survey. J. Syst. Software, 70: 
61-78. 
Fracker, M. L. (1988). A theory of situation assessment: implications for measuring situation 
awareness. In Proceedings of Human Factors Society 32nd Annual Meeting (pp. 
102-106). 
247 
 
Fracker, M. L. (1991). Measures of situation awareness: Review and future directions (Rep. 
No.AL-TR-1991?0128). Wright Patterson Air Force Base, Ohio: Armstrong 
Laboratories, Crew Systems Directorate. 
Gao, T., Massey, T., Selavo, L., Welsh, M., & Sarrafzadeh, M., (2007). Participatory user 
centred design techniques for a large scale ad-hoc health information system. In: Proceedings 
of 1st ACM SIGMOBILE International Workshop on Systems and Networking 
Support for Healthcare and Assisted Living Environments (Healthnet 07 in 
Conjunction with MobiSys? 07). Pureto Rico, USA, 11 June, 2008, pp. 43-48.  
Gibbins, N., Harris, S., & Shadbolt, N. R. (2003) Agent-based semantic web services. Paper 
presented at the 12th International World Wide Wed Conference (WWW2003), 
Budapest, Hungary. 
Grant, T., (2009). Towards mixed rational-naturalistic decision support for command and control. 
Proceedings of 6th International Information Systems Crisis Management 
Systems (ISCRAM) Conference, Gothenburg, Sweden.  
Gregg, D., Kulkarni, U., & Vinze, A. (2001). Understanding the philosophical 
underpinnings of software engineering research in information 
systems. Information Systems Frontiers 3(2): 169-183.  
Gruber, T.R. (1993) A translation approach to portable ontology specifications. 
Knowledge Acquisition 5 (2): 199?220, 1993. Available at: 
http://tomgruber.org/writing/ontolingua-kaj-1993.pdf 
Guarino, N., (1998). Formal ontology and information systems. Amended version of paper 
appeared in N. Guarino (ed), Formal Ontology in Information Systems. 
Proceedings of FOIS?98, Trento, Italy, 6-8 June, 1998, Amsterdam, IOS Press, 
pp. 3-15.  
Guba, E., & Lincoln, Y. (1994). Competing paradigms in qualitative research. The Handbook 
of Qualatative Research. N. Denzin and Y. Lincoln. Thousand Oaks, CA, 
Sage: 105-117. 
Gugerty, L. (1998). Evidence from partial report task for forgetting in dynamic spatial 
memory. Human Factors, 43(3). 498-508. 
248 
 
Gugerty, L., Tirre, W. (1996). Situation awareness: a validation study and investigation of 
individual differences. Proceedings of Human Factors and Ergonomics Society 40th 
Annual Meeting. 40(12). pp. 564-568.  
Gunnarsson, B. M., & Stomberg, M. W. (2008). Factors influencing decision making 
among ambulance nurses in emergency care situations. International Emergency 
Nursing, 83-89 
Haddow, G., Bullock, J., & Coppola, D. (2007). Introduction to emergency management, 
3rd Edition. Elsevier Press Homeland Security Series. Butterworth-Heinemann. 
Burlington. 
Hamm, R. M. (1988) Clinical intuition and clinical analysis: expertise and the cognitive 
continuum. In Professional Judgement: A Reader in Clinical Decision Making (Dowie J 
& Elstein A eds). Cambridge University Press, Cambridge. 
Hammersley, M., & Atkinson, P. (1995). Ethnography: Principles in practice, 2nd Edn. 
London: Routledge.  
Hauland, G. (2008). Measuring individual and team situation awareness during planning 
tasks in training of en-route air trafic control. The International Journal of Aviation 
Psychology, 18: 3, 290 ? 304.  
Hedberg, B., & Larson, U. S. (2004). Environmental elements affecting the decision 
making process in nursing practice. Journal of Clinical Nursing 13, 316?324. 
Hernandez, J. Z., & Serrano, J. M. (2001). Knowledge-based models for emergency 
management systems. Expert Systems with Applications, 20,173-186 
Hevner, A., & Chatterjee, S. (2010). Design science research in information systems: 
Theory and practice. Springer, New York Dordrecht Heidelberg London. ISBN 
978-1-4419-5652-1. DOI 10.1007/978-1-4419-5653-8.  
Hevner, A., March, S., Park, J., & Ram, S. (2004). Design sceince in information systems 
research. MIS Quaterly 28, 75-105.  
Hinsley, D., Hayes, J.R., Simon, H.A. (1977). From words to equations. In Proceedings of 
Cognitive processes in Comprehension. Hillsdale, N.J. Erlbaum.  
249 
 
Hoffman, R. R., Crandall, B., & Shadolt, N. (1998). Use of critical decision method to 
elicit expert knowledge: a case study in the methodology of cognitive task 
Analysis. Human Factors, 40(2), pp. 254-276.  
Hoffman, R. R., & Militello, L. G. (2008). Perspectives on cognitive tasks analysis: Historical 
origins and modern communities of practice. NY: Psychology Press, Taylor & Francis 
Group.  
Hoffman, R. R., & Woods, D. D. (2000). Studying cognitive systems in context: Preface 
to the special section. Human Factors, 42(1), 1-7.  
Hoogendoorn, M., Jonker, C. M., Popova, V., Sharpanskykh, A., & Xu, L. (2005). 
Formal modelling and comparing of disaster plans. Paper presented at the Proceedings 
of the Second International Information Systems for Crisis Response and 
Management (ISCRAM), Brussels, Belgium. 
Horney, T., Jungert, E. & Folkesson, M. (2003). An ontology controlled data fusion process for 
a query language. In: Sixth International Conference of Information Fusion, pp. 
530-537. 
Hough, P. (2008). Understanding global secuirty. Taylor & Francis, 2008. ISBN 0415421411, 
9780415421416 
Huang, T., Li, T., and Yang, C. (2008). Comparison of ontology reasoners: Racer, Pellet, Fact++. 
AGU Fall Meeting absracts. abstract #IN13A-1068.  
Hutchins, E. (1995). How a cockpit remembers its speeds. Cognitive Science. 19, 265-288.  
Hutton, R. J., & Klein, G. (1999). Expert decision making. System Engineering, 2(1), 32-45. 
Hwang, Z., Uhm, Y., Lee, M., Kim, G., & Park, S. (2007). A context aware system 
architecture for dealing with the emergency by the community service in apartment. In: 
Proceedings of the Future Generation Communication And Networking 
(FGCN 2007). Jeju Island, Korea, 06-08 Dec, 2007, pp. 402-407.  
Iakovou, E., & Douligeris, C. (2001). An information management system for the 
emergency management of hurricane disasters. International Journal of Risk 
Assessment and Management, 2(3-4), 243-262.  
250 
 
Iivari, J. (2007). A paradigmatic analysis of information systems as a design science. 
Scandinavian Journal of Information Systems, 19(2), Article 5.  
ISO/IEC FCD 9126-1, (1998). Software product quality - Part 1: Quality model, 1998. 
Jang, H. C., Lien, Y. N., & Tsai, T. C. (2009). Rescue information system for earthquake 
disasters based on MANET emergency communication platform, In: Proceedings of the 
International Workshop on Advanced Topics in Mobile Computing for 
Emergency Management: Communication and Computing Platforms (MCEM 
2009). Leipzig, Germany, 21-24 June, 2009, pp.623-627.    
J?rvinen, P. (2007). Action research is similar to design science, Quality & Quantity, 41, 
37-54. 
Javed, Y., & Norris, A. C. (2011). Prototyping and evaluation of a computerised emergency 
management system based on an ontological inference design. In proceedings of IEEE 
Internal Global Humanitarian Conference 2011, Seattle, Washington, USA.  
Javed, Y., Norris, A. C., & Johnston, D. M. (2011a). Mass evacuation support system 
(MESS): Validating information requirements of emergency managers. In Proceedings of 
7th International Conference on Information Technology and Applications 
(ICITA 2011). Sydney, Australia. 
Javed, Y., Norris, A. C., Johnston, D., & Doyle, E. H. (2011b). Towards a framework 
for crisis decision support systems: Information requirements for contextual 
team situation awareness In: CutterIT Journal: Special Issue on IT and Crisis 
Management, 24(1), 26-33.  
Jennex, M. E. (2004). Emergency response systems: the utility Y2K experience, Journal of 
IT Theory and Application, 6(3), 85-102.  
Jennex, M. E. (2007). Knowledge management in modern organizations. San Diego State 
University, USA. DOI: 10.4018/978-1-59904-261-9, ISBN13: 9781599042619.  
Jia. X.N., Rong. L.L. (2006). Classification based management method for government 
documents in emergency response. KSS?2006. Beijing, China.  
Jones, C. (1994). Assessment and control of software risks, Prentice Hall, ISBN 0137414064.  
251 
 
Jones, D. G., & Endsley, M. R. (1996). Sources of situation awareness errors in aviation. 
Aviation, Space, Environmental Medicine, 67(6), 507-512). 
Jones, D. G., & Endsley, M. R. (2002). Measurement of shared SA in teams: In initial 
investigation (No. SATech-02-05). Marietta, GA: SA Technologies 
Kaempf, G.L., Wolf, S., Miller, T.E. (1993). Decision making in AEGIS combat information 
centre. In proceedings of Human Factors and Ergonomics Society 37th Annual 
Meeting. (pp. 1107-1111). 
Kanchanasut, K., Tunpan, A., Awal, M., Das, D., Wongsaardsakul, T. & Tsuchimoto, T. 
(2007). DUMBONET: a multimedia communication system for collaborative 
emergency response operations in disaster-affected areas, International Journal of 
Emergency Management, 4(4), 670-681.  
Kaplan, C., & Simon, H. A., (1990). In search of insight. Cognitive Psychology, 22(3), 374-
419.  
Katuk, N., Ku-Mahamud, K.R., Norwawi, N. & Deris, S., (2009). Webs-based support 
system for flood response operation in Malaysia. International Journal of Disaster 
Prevention and Management, 18(3), 327-337.  
Keina G & Friedland N (1984) Dilemmas concerning the training of individuals for task 
performance under stress. Journal of Human Stress 10, 185?190. 
Kirlik, A., & Strauss, R. (2006). Situation awareness as judgment I: Statistical modeling 
and measurement. International Journal of Industrial Ergonomics: Special issue on 
?Cognitive engineering insights for human performance and decision making.?  36. 463-474. 
Kirwan, B., & Ainsworth, L. K. (eds.), (1992). A guide to task analysis.London: Taylor & 
Francis, Ltd. 
Klein, G. (1989). Recognition-primed decisions. In Advances in Man Machine Systems 
Research (Rouse WB ed). JAI Press, Greenwich, Connecticut. 
Klein, G. (1993). A recognition primed decision (RPD) model of rapid decision making. In 
Decision making in action: models and methods (Klein G, Orasanu J, 
Calderwood R & Zsambok C eds). Ablex Publishing, Norwood, New Jersey, pp. 
138?147. 
252 
 
Klein, G. (1997), "Recognition-primed decision making", in Zsambok, C., Klein, G. 
(Eds), Naturalistic Decision Making, Lawrence Erlbaum, Hillsdale, NJ, .  
Klein, G. (1998). Sources of power: How people make decisions. Cambridge, Massachusetts: 
MIT Press, p3.  
Klein, G. (1999). Sources of power, Cambridge, MA: MIT Press: Cambridge.  
Klein, G. (2000). Analysis of situation awareness from critical incident report. In: M. R. 
Endsley & D. J. Garland, eds. Situation Awareness Analsysis and Measurement. 
Mahwa, NJ: Lawrence  Erlbaum Associates Inc., pp 51-72.  
Klein, G., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid decision making on the fire 
ground. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 
576-580). 
Klein, G., & Klinger, D. (1991) Naturalistic decision making. Human Systems. IAC 
Gateway, Vol XI No 3. Available online: 
http://WWW.au.af.mil/au/awc/awcgate/decision/nat-dm.pdf Accessed: 27 
July 2009) 
Klein, G., Orasanu, J., Calderwood, R., and Zsambok, C.E. (1993) decision making in 
action: Models and methods. Ablex Publishing Co., Norwood, NJ 
Knnab, R. D., Rhome, J. R., & Brown, D. P. (2011). Tropical cyclone report hurricane Katrina, 
2005. Updated 14 September 2011 to include damage estimates from the 
National Flood Insurance Program and to revise the total damage estimate. 
Available at: http://www.nhc.noaa.gov/pdf/TCR-AL122005_Katrina.pdf 
Kobus, D. A., Proctor, S., & Holste, S. (2001). Effects of experience and uncertainty 
during dynamic decison making. International Journal of Industrial Ergonomics, 28(5), 
275-290.  
Kokar, M. M., Matheus, C. J., Baclawski, K., Letkowski, J. A., Hinman, M., & Salerno, J. 
(2004). Use cases for ontologies in information fusion. In Proceedings of the Seventh 
International Conference on Information Fusion, Stockholm, Sweden.  
253 
 
Kontopoulos, E., Zetta, T., & Bassiliades, N. (2012). Semantically-enhanced authoring of 
defeasible logic rule bases in the semantic web. In proceedings of the 2nd International 
Conference on Web Intelligence. Article No. 56. ISBN: 978-1-4503-0915-8 
Kwan, M. P., & Lee, J. (2005). Emergency response after 9/11: The potential of real 
time 3D GIS for quick emergency response in micro-spatial environments. 
Computers, Environment and Urban Systems, 29(2).   
Lai, C. H., Yang, J. C., Chen, F. C., Ho, C. W., & Chant, T. W. (2007). Affordances of 
mobile technologies for experiential learning: the interplay for technology and 
pedagogical practices. Journal of Computer Assisted Learning, 326-337. 
Lakatos, I. (1978). The methodology of scientific research programmes: Philosophical Papers 
Volume 1. Cambridge: Cambridge University Press.  
Lanfranchi, V. & Ireson, N., (2009). User requirements for collective intelligence emergency 
response system. In: Proceedings of the 2009 British Computer Society Conference 
on Human Computer Interaction. Cambridge, United Kingdom, 1-5 Sept, 2009, 
pp. 198-203. 
Laudan, L. (1981). Science and hypothesis. Dordrecht, the Netherlands: Reidel. 
Lee, J., & Bui, T. (2000). A template-based methodology for disaster management information 
systems. In: Proceedings of 33rd Hawaii International Conference on System 
Sciences (HICSS). Maui, Hawaii, 4-7 Jan, 2000.  
Lien, Y. N., Jang, H. C., & Tsai, T. C. (2009). A MANET based emergency communication 
and information system for catastrophic natural disasters. In: Proceedings of the 29th 
IEEE International Conference on Distributed Computing Systems Workshops. 
Montreal. Quebec, Canada, 22-26 June 2009, pp. 412-417.  
Little, E. G., & Rogova, G. L. (2005). Ontology meta-model for building a situational picture of 
catastrophic events. Paper presented at the 2005 8th International Conference on 
Information Fusion. 
Liu, W.-Y., & Fang, K. (2006). Using IDEF0/Petri net for ontology-based task knowledge 
analysis: The case of emergency response for debris-flow. System Sciences, 2006. HICSS 
254 
 
'06. Proceedings of the 39th Annual Hawaii International Conference 04-07 Jan, 
2006, ISSN: 1530-1605.  
Liu, K.F.R. (2004). Agent-based resource discovery architecture for environmental 
emergency management. Expert Systems with Applications, 27, 77?95 
Lorincz, K., Malan, D.J., Jones, T.R.F.F., Nawoj, A., Clavel, A., Shnayder, v., Mainland, 
J., Welsh, M., & Moulton, S. (2004). Sensor networks for emergency response: 
challenges and opportunities. Pervasive Computing, 3(4), pp. 16-23. 
Luce, R. D., & Winterfeldt, D. (1994). What common ground exists for descriptive, 
prescriptive, and normative utility theories? Management Science, Vol. 40, No. 2 
(Feb., 1994), 263-279 
Madey, G.T., Szabo, G. & Barabasi, A.L., (2006). WIPER: The integrated wireless phone 
based emergency response system. In: Proceedings of the 6th International Conference 
on Computational Science (ICCS 2006). Reading UK, 28-31 May, 2006, pp. 417-
424. 
Maguire, M. (2001a). Methods to support human centred design. International Journal of 
Human Computer Studies, 55(4), 587-634.  
Maguire, M. (2001b). Context of use within usability activities. International Journal of 
Human Computer Studies 55(4), 453-483.  
Manoj, B. S., & Baker, A. H., (2007). Communication challenges in emergency response. 
Communications of the ACM, 50(3), 51-53.  
Martin, D., Burstein, M., Hobbs, J., Lassila, O., McDermott, D., McIIraith, S., 
Narayanan, S., Paulucci, M., Parsia, B., Payne, T., Evren, S., Naveen, S., & 
Sycara, K. (2004). OWL-S: Semantic markup for web services. World Wide Web 
Consortium: http://www.w3.org/Submission/2004/SUBM-OWL-S-20041122/ 
March, S.T., & Smith, G. F. (1995). Design and natural science research on information 
technology. Decision Support Systems, 15, 251-266. 
Massey, T., Gao. T., Welsh, M., Sharp, J., & Sarrafzadeh, M. (2006). The design of a 
decentralised electronic triage system. In: Proceedings of Annual Symposium of 
255 
 
American Medical Informatics Association (AMIA 2006). Washington, DC, 11-
15 Nov. 2006, pp 544-548. 
Matheus, C., Kokar, M., & Baclawski, K. (2003) A core ontology for situation awareness.Paper 
presented at the 6th International Conference on Information Fusion, Cairns, 
Queensland, Australia. 
Matheus, C., Kokar, M., Baclawski, K., Letkowski, J., & Call, C. (2004) The practical 
application of semantic web technologies for situation awareness. Paper presented at the 
International Semantic Web Conference, Hiroshima, Japan. 
Matheus, C. (2005) Using ontology-based rules for situation awareness and information fusion. 
Paper presented at the W3C Workshop on Rule Languages for Interoperability, 
Washington D.C., USA. 
MCDEM ? Ministry of Civil Defence and Emergency Management (2008a). Mass 
evacuation planning, director?s guideline for civil defence emergency management groups. 
Available at:http://www.pacificdisaster.net/pdnadmin/data/original/DGL_07-
8_Mass_evacuation_planning.pdf 04-Full_Report.pdf. Last accessed 24 March 
2011. 
MCDEM ? Ministry of Civil Defence and Emergency Management (2008b). Exercise 
Ruaumoko 08: Final exercise report. Available at: 
http://www.civildefence.govt.nz/memwebsite.nsf/Files/National%20Exercise
%20Programme/$file/ExRuaumoko-FINAL-REPORT-Aug08.pdf. Last 
accessed  27 May, 2012.   
MCDEM ? Ministry of Civil Defence and Emergency Management (2009). The guide to 
the national civil defence emergency management plan. Revised in 2009. Available at: 
http://reliefweb.int/sites/reliefweb.int/files/resources/0BC61CC65F72B74E4
92577C90025E804-Full_Report.pdf. Last accessed 20 March 2011. 
MCDEM ? Ministry of Civil Defence and Emergency Management (2011). Exercise 
Tangaroa 2010: Final exercise report. Available at: 
http://www.civildefence.govt.nz/memwebsite.nsf/Files/Exercise-
Tangaroa/$file/Tangaroa-final-reprt-10Aug11.pdf. Last accessed  27 May, 2012.   
256 
 
McGrath, S., Grigg, E., Wendelken, S., Blike, G., de-Rossa, M., Fiske, A., & Gray, R. 
(2003). ARTEMIS: A vision for remote triage and emergency management information 
integration. Available at: http://www.ists.dartmouth.edu/library/15.pdf.  
Accessed on August 14, 2011. 
McGuinness, D. L. (2003). Ontologies for information fusion. In: Sixth International 
Conference on Information Fusion, pp. 650-656. 
Mcllvaine, W. B. (2007). Situation awareness in operating room. A Primer for the 
Anaesthesiologist. Journal of Critical Care, 26 (3), 167-172.  
Meissner, A., Luckenbach, t., Risse, T., Kirste T., & Kirchner, H. (2002). Design challenges 
for an integrated disaster management communication and information system. In: 
Proceedings of the 1st IEEE Workshop on Disaster Recovery Networks 
(DIREN 2002).   
Meissner, A., Zhou, W., Putz, W. & Grimmer, J., (2006). MIKoBOS ? A mobile information 
and communication system for emergency response. In: Proceedings of the 3rd 
International ISCRAM Conference, Newark, NJ, USA, 14-17 May 2006, pp. 92-
101.  
Mendonca, D., Rush, R., & Wallace, W.A. (2000). Timely knowledge elicitation from 
geographically separate, mobile experts during emergency response. Safety Science, 
35,193-208 
Militello, L. G., & Hutton, R. J. B. (1998). Applied cognitive task analysis (ACTA): A 
practitioner?s toolkit for understanding cognitive task demands. Ergonomics, Vol. 
41(11), 1618 ? 1641. 
Mirfenderesk, H. (2009). Flood emergency management decision support system on the 
Gold Coast, Australia. The Australian Journal of Emergency Management. 
Mitroff, I. I., & Linstone, H. A. (1993). The unbounded mind: breaking the chains of traditional 
business thinking. (New York: Oxford University Press) (1993), ISBN 
0195077830. 
Mogofrd, R. H. (1997). Mental models and situation awareness in air traffic control. 
International Journal of Aviation Psychology, 7(4), 331-342.  
257 
 
National Research Council, (2011). "Summary" Tsunami warning and preparedness: An 
assessment of the U.S. tsunami program and the nation's preparedness efforts. Washington, 
DC: The National Academies Press, 2011. ISBN 978-0-309-13753-9 
 Newell, A. (1990). Unified theories of cognition. Harvard University Press, Cambridge, 
Massachusetts.  
Niculae, C., French, S., & Carter, E. (2004). Emergency management: Does it have a 
sufficient comprehensive understanding of decision making process and 
context? Radiation Protection Dosimetry , 97-100. 
Nofi, A. (2000). Defining and measuring shared situational awareness, DARPA. Available from: 
http://www.thoughtlink.com/publications/DefiningSSA00Abstract.htm 
Normile, D. (2011). Devastating earthquake defied exptectations. (Science Mag) Retrieved 
September 12, 2011, from sciencemag.org 
Noy, N. F., & McGuinness, D. L. (2001). Ontology development 101: A guide to creating your 
first ontology. Knowledge Systems Laboratory, March, 2001. Available at: 
http://www.ksl.stanford.edu/KSL_Abstracts/KSL-01-05.html 
O'Connor, P., Campbell, J., Newon, J., Melton, J., Salas, E., & Wilson, K. (2008). Crew 
resource management training effectiveness: A Meta-Analysis and Some Critical 
Needs. International Journal of Aviation Psychology, 18, 4, 353-368. 
Orasanu, J. (1997). Stress and naturalistic decision making: strengthening the weak links, 
in Flin, R., Salas, E., Straub, M., Martin, L. (Eds),Decision Making under Stress: 
Emerging Themes and Applications, Ashgate, Aldershot. 
Orasanu, J., & Connolly, T. (1993). The reinvention of deciosn making. In: G. Klein, J. 
Orasanu, R. Calderwood & C. Zsambok, eds. Decision making in Action, 
Models and Methods. Norwood, NJ: Albex Publishing Corporation. pp. 3-20.   
Otim, S. (2006). A case-based knowledge management system for disaster management: Fundamental 
concepts. In: Proceedings of the 3rd International ISCRAM Conference, Newark, 
NJ, USA. 14-17 May, 2006, pp. 598-604. 
Padmanabhan, N., Burstein, F., Churilov, L., Wassertheil, J., Hornblower, B., & Parker, 
N. (2006). A mobile emergency triage decision support system evaluation. In Proceedings 
258 
 
of the 39th Annual Hawaii International Conference on Volume 5, Issue, 04-07 
Jan. 2006.  
Paton, D., Johnston, D., Houghton, B., Flin, R., Ronan, K., & Scott, B. (1999). 
Managing natural hazard consequences: information management and decision 
making, Journal of the American Society of Professional Emergency Managers, Vol. 6 37-
48.  
Paton, D., & Jackson, D. (2002). Developing disaster management capability: an 
assessment centre approach. Disaster Prevention and Management, 115-122. 
Peffers, K., Tuuananen, T., Rothenberger, M., & Chatterjee, S. (2008). A design science 
research methodology for information systems research. Journal of Management 
Information Systems. 24(3), 3, 45-77.  
Perla, P., Markowitz, M., Nofi, A., Weuve, C., Loughran, J., & Stahl, M. (2000). Gaming 
and  shared  situation  awareness,  DARPA.  Available  from:  
http://www.thoughtlink.com/ publications/GamingSSA00Abstract.html 
Pfaff, M. S., Drury, J. L., Klein, G. L., More, L., Moon, S. P., & Liu, Y. (2010). Weighing 
decisions: Aiding emergency response decision making via option awareness. 2010 IEEE 
International Conference on Technologies for Homeland Security, HST 2010. 
pp 251 ? 257.  
Potter, S., Kalfoglou, Y., Alani, H., Bachler, M., Shum, S.B., Carvalho, R., Chakravarthy, 
A., Chalmers, S., Chapman, S., Hu, B., Preece, A., Shadbolt, N., Tate, A., & 
Tuffield, M. (2007). The application of advanced knowledge technologies for emergency 
response. In: Proceedings of the 4th International ISCRAM Conference, Delft, 
Netherlands, 13-16 May 2007, pp. 361-368. 
Prasanna, R., Yang, L. & King, M., (2009). GDIA: A cognitive task analysis protocol to capture 
the information requirements of emergency first responders. In: Proceedings of the 6th 
International ISCRAM Conference. Gothenburg, Sweden, 10-13 May 2009.    
PSC - Public Safety Canada (2011).  An emergency management framework for Canada,  Second 
Edn. Available at: http://www.publicsafety.gc.ca/prg/em/_fl/emfrmwrk-2011-
eng.pdf Accessed 20 March 2011.  
259 
 
Quarantelli, E. L. (1998). What is a disaster? Perspectives on the question. London: Routledge.  
Ramrekha, T.P. & Politis, C., (2009). An adaptive QoS solution for MANET based 
multimedia communications in emergency cases. In: Proceedings of the 1st 
International ICST Conference (MOBILIGHT 2009). Athens, Greece, 18-20 
May, 2009, pp. 74-78.  
Rekimoto, J. (1996). Transvision: A hand-held augmented reality system for collaborative 
design.  International Conference on Virtual Systems and Multimedia 
(VSMM?96), pp. 85-90.  
Resnick, M. L. (2003). Situation awareness applications to executiv dashboard design. Human 
Factors and Ergonomics Society 47th AnnualConference, Human Factors and 
Ergonomics Society.  
Reyna. V. F., & Rivers, S. E. (2009). Current theories of risk and rational decision 
making.  Dev Rev. PubMed. doi: 10.1016/j.dr.2008.01.002 
Riley, J.M., Endsley, M. R., Bolstad, C. A., Cuevas, H. M. (2006). Collaborative planning 
and situation awareness in army command and control. Ergonomics, 49, 1139?
1153. 
Rittel, H., & Webber, M. M. (1984). Planning problems are wicked problems. In N. Cross 
(ed.), Developments in Design Methodology, John Wiley & Sons, New York, 
1984, pp. 135-144.  
Robinson, W. N., & Elofson, G. (2004). Goal directed analysis with use cases. Journal of 
Object Technology, Volume 3, no. 5 (May 2004), 125-142.  
Rodgers, M. D., Mogford, R. H., & Strauch, B. (2000). Post hoc assessment of situation 
awareness in air traffic control incidents and major aircraft accidents. In D. J. 
Rong, L. L. (2005). Reorganizing the knowledge in government documents for quick response. KSS? 
2005. Laxenburg, Austria  
Rogalski, J., Samurcay, R. (1993), Analysing communication in complex distributed 
decision making", Ergonomics, Vol. 36 pp.1329-42. 
260 
 
Rogova, G.L., Scott, P.D., Lollett, C., & Mudiyanur, R. (2006). Reasoning about situations in 
the early post disaster response environment. In FUSION?06, 9th International 
Conference on Information Fusion, contribution No. 211. Florence, Italy, 2006.  
Rosen, J., Grigg, E., Lanier, J., MacGrath, S., Lillibridge, S., Sargent, D., & Koop, C.E. 
(2002). The Future of the Command and Control for Disaster Response. IEEE 
Engineering in Medicine and Biology Magazine, 21(5),56-68. 
Rosenthal, U., Boin, A., & Comfort, L. K., (2001). Managing crises: Threats, dilemmas, 
opportunities. Springfield, IL: Charles C. Thomas Publisher Ltd.  
Roth E. M. (1997). Analysis of decision making in nuclear power plant emergencies: An 
investigation of aided decision making. In: J. J. Marciniak, ed. Encyclopedia of 
Software Engineering. 2nd  ed. NY: John Wiley and Sons Inc., pp. 163 ? 179.  
Said, A. M., Ahmadun, F.-R., Mahmud, A. R., & Abas, F. (2011). Community 
preparedness for tsunami disaster: case study. Disaster Prevention and Management, 
266-280. 
Salas E, Prince, C., Baker, D. P., & Shrestha, L. (1995). Situational awareness in team 
performance: implications for measurement and training. Human Factors 37, 123?
136. 
Salas, E., (2005), Team methods. In handbook of human factors methods, N.A. Stanton, A. 
Hedge, K. Brookhuis, E. Salas and H. Hendrick (Eds) (Boca Raton, FL: CRC 
Press). 
Salmon, P. M., Stanton, N. A., Walker, G. H., Baber, C., Jenkins, D. P., Mcmaster, R., et 
al. (2008). What really is going on? Review of situation awareness models for 
indivduals and teams. Theoretical Issues in Ergonomics Science, 9(4), 297-323. 
Salmon, P. M., Stanton, N. A., Walker,G. H., Jenkins, D., Ladva, D., Rafferty, L., and 
Young, M. (2009). Measuring situation awareness in complex systems: 
Comparison of measures study, International Journal of Industrial Ergonomics, 
Volume 39, Issue 3, May 2009, 490-500. 
261 
 
Saner, L. D., Bolstad, C. A.,Gonzales, C., & Cuevas, H. M. (2009). Measuring and 
predicting shared situation awareness in teams. Journal of Cognitive Engineering and 
Decision Making, Volume 3. 
Sarter, N. B., & Woods, D. D. (1991). Situation awareness: A critical but ill-defined 
phenomenon. The international Journal of Aviation Psychology, 1(1), 45-57.  
Satake, K., Rabinovich, A., Kanoglu, U., & Tinti, S. (2011). Introduction to "Tsunamis 
in the World Ocean: Past, Presnt and Future. Volume I". Pure Applied Geophysics, 
963-968. 
Sauro, J., & Kindlund, E. (2005). A method to standardize usability metrics into a single 
score. Proceedings of CHI 2005, New York, NY: ACM. pp. 401-409.. 
Schilit, B. N. (1995). A context-aware system architecture for mobile distributed 
computing. Doctoral dessertation , Columbia University, May 1995. 
Schilit, B. N., Adams, N., & Want, R. (1994). Context-aware computing applications. Palo 
Alto, Calif: Xerox Corp., Palo Alto Research Center.  
Secrist, G.E., Hartman, B. O. (1993). Situation awareness: The trainability of near-
threshold information acquisition dimension. Aviation, Space and Environmental 
Medicine, 64, 885-892. 
Shadbolt, N., Berners-Lee, T. and Hall, W. (2006) The semantic web revisited, IEEE 
Intelligent Systems, 21(3), 96?101.  
Shu, Y., & Furuta, K. (2005), An inference method of team situation awareness based 
on mutual awareness. Cognition Technology & Work, 7, 272?287. 
Simon, H. A. (1955). A behavioural model of rational choice. The Quarterly Journal of 
Economics, 69(1), 99-118. 
Simon, H. A. (1989). Decision making in organisations. In Writers on Organisations, 4th 
edn (Pugh DS & Hickson DJ eds). Penguin, London.  
Simon, H. A. (1996). Sciences of the Artificial, Cambridge: MIT Press.  
Sinha, R. (2005). Impact of experience on decision making in emergency situation. Extended Essay. 
Psychology C/D. Sweden: Lulea University of Technology, Sweden.  
262 
 
Smallman, C., & Weir, D. (1999). Communication and cultural distortion during crises. 
Disaster Prevention and Management, 8(1) 33-41. 
Smart, P. R., Shadbolt, N. R., Carr, L. A., & schraefel, M. C. (2005). Knowledge-based 
information fusion for improved situational awareness. Paper presented at the 8th 
International Conference on Information Fusion, Philadelphia, USA. 
Smirnov, A., Levashova, T., & Shilov, N. (2010). Ubiquitous computing in emergency: Profile 
based situation response. In: Seventh International ISCRAM Conference, Seattle, 
pp.  1-10, 2010.  
Smirnov, A., Pashkin, M., Chilov, N., & Levashova, T. (2003). KSNet-Approach to 
knowledge fusion from distributed sources. Computing and Informatics, 22: 105?
142, 2003.   
Smith, A., & Russell, J. (1991). Using critical incidents in nurse education. Nurse 
Education Today 11, 284?291. 
Smith, K., Hancock, P.A. (1994). Situation awareness is adaptive, externally-directed 
consciousness. Situational awareness in complex systems. (59-68). Daytona Beach, FL: 
Embry-Riddle Aeronautical university Press. 
Smith, K., Hancock, P.A. (1995). Situation awareness is adaptive, externally directed 
consciousness. Human Factors, 37(1), (137-148). 
Snowden, D. (2002). Complex acts of knowing paradox and descriptive self-awareness. J 
Knowl. Manage. 6(2),100-111 (2002). 
Son, J., Aziz, Z., & Pena-Mora, F. (2007). Supporting disaster response and recovery 
through improved situation awareness. Structural Survey, 26 (5), 411-425.  
Stanton, N.A., Chambers, P. R. G., Piggott, J. (2001). Situational awareness and aafety. 
Safety Science, 39 (3). pp. 189-204. 
Stanton, N.A., Salmon, P. M., Walker, g. H., Baber, C., Jenkins, D. P. (2005). Human 
factors methods: A practical guide for engineering and design. Aldershot, UK: Asgate 
Publishing Ltd. 
263 
 
Sternberg, R. J., Forsythe, G. B., Hedlund, J., Horvath, J. A., Wagner, R. K., Williams, 
W. M., Snook, S. A., & Grigorenko, E. L. (2000), Practical intelligence in everyday life, 
Cambridge University Press, Cambridge. 
Stevens, M. J., & Campion, M. A. (1994). The knowledge, skill and ability requirements 
for teamwork: implications for human resource management. Journal of 
Management  20, 503?530. 
Strickland, J. (2008). How cloud computing works. HowStuffWorks. Available at: 
http://computer.howstuffworks.com/cloud-computing/cloud-computing.htm 
Last accessed on June 10, 2012. 
Studer, R., Benjamins, V. R., & Fensel, D. (1998). Knowledge engineering: principles 
and methods. Data and Knowledge Engineering, 25(1-2):161?197.  
Sycara, K., Paulucci, M., & Lewis, M. (2003) Information discovery and fusion: semanticson the 
battlefield. Paper presented at the 6th International Conference on Information 
Fusion, Cairns, Queensland, Australia. 
Taylor, R.M., (1990). Situational awareness rating technique (SART): The development of a tool for 
aircrew systems design. In Situational Awareness in Aerospace Operations (AGARD-
CP-478) pp3/1 ?3/17, Neuilly Sur Seine, France: NATO-AGARD. 
Taylor, R.M., Endsley, M.R., Henderson, S. (1996). Situation awareness workshop 
report. Applied aviation psychology: Achievment, change and challenge, (447-454). 
Aldershot, UK: Ashate Publishing Ltd.  
Thompson, C., & Dowding, D. (2002). Clinical decision making and judgement in nursing. 
Churchill Livingstone, Edinburgh. 
Townsend, A., DeMarie, S., and Hendrickson, A. 1998. Virtual teams: Technology and 
the workplace of the future. Academy of Management Executive, 12(3):17 ? 29. 
Tran, T., Lewen, H., & Haase, P. (2007). On the role and application of ontology in information 
systems. In: 2007 IEEE International Conference on Research Innovation and 
Vision for the Future. pp. 14-21. ISBN: 1424406943. 
DOI: 10.1109/RIVF.2007.369130, 2007.  
264 
 
Tullis, T., & Albert, W. (2008). Measuring the user experience: Collecting, analyzing and 
presenting usability metrics. Boston: Morgan Kauffman/Elsevier. 
Turoff, M. (2002) Past and future emergency response information systems. 
Communications of the ACM, 45(4), 29-33.  
Turoff, M., Chumer, M., Van de Walle, B., & Yao, X., (2004). The design of a dynamic 
emergency response management information system (DERMIS). Journal of 
Information Technology Theory and Application (JITTA), 5(4), 1-35.  
Uhlarik, J., & Comerford, D. A. (2002). A review of situation awareness literature relevant to 
pilot surveillance functions. Washington, DC: FAA Office of Aerospace Medicine, 
Civil Aerospace Medicine Institute. 
Uschold, M., & Gr?ninger, M. (1996). Ontologies: Principles, methods, and 
applications. Knowledge Engineering Review, 1:96-137. 
Usability Professional?s Association, (2011). What is user centred design? Available at: 
http://www.upassoc.org/usability_resources/about_usability/what_is_ucd.html 
Accessed on July, 8, 2011.  
Vaishnavi, V. K., & Kuechler, W. (2008). Design science research methods and patterns. 1st 
Edn., Auerbach Publications, Taylor and Francis Group, Boca Raton, FL., 
ISBN: 978-1-4200-5932-8. 
Van de Walle, B., & Turoff, M. (2007). Emergency response information system: 
Emerging trends and technology. Communications of ACM, 50(3), 29-31. 
Vicente, K. J. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based 
work. Mahwah, NJ: Lawrence Erbaum Associates.  
Vidulich, M. A. (2000). Testing the sensitivity of situation awareness Metrics in interface 
evaluations. In Situation Awareness Analysis and measurement, Taylor & Francis. 
Viinikkala, M. (2004). Ontology in information systems. 8108103. Available at: 
http://www.cs.tut.fi/kk/webstuff/Ontology.pdf. 
Wagner, R. K., & Sternberg, R. J. (1985), Practical intelligence in real-world pursuits: the 
role of tacit knowledge. Journal of Personality and Social Psychology, 49, 436-58.  
265 
 
Wang, Q., & Rong, L. (2007). Agile knowledge supply chian for emergency decsion 
making support. ICCS Springer-Verlag Berlin , 178-185. 
Wang, W., Zhong, S., Zhang, Z., Senlin, L., & Wang, L. (2009). Empirical research and 
design of m-learning system for college english. In M. Chang et 
al.(Eds.), Learning by Playing. Game-based Education System Design and 
Development  56(70), 524-535.  
Weidenhaupt, K., Pohl, K., Jarke, M., Haumer, P., & Aachen, T.H. (1998). Scenarios in 
system development: Current practice. Software, IEEE, 15: 34-45. 
Wellens, A. R. (1989). Effects of telecommunication media upon information sharing 
and team performance: Some theoretical and empirical observations. IEEE 
Aerospace and Electronic Systems Magazine, New York: The Institute of Electrical 
and Electronics Engineers, Inc. 4, 13 -19.  
Wellens, A.R., (1993), Group situation awareness and distributed decision making: From 
military to civilian applications. In Individual and Group Decision Making: Current 
Issues, N.J. Castellan (Ed.), pp. 267?287 (Mahwah, NJ: Erlbaum Associates). 
Wenjun, W., Yuelong, W., Fankuo, M., Wei, Y., Yingwei, L., & zhuoqun, X. (2005). The 
semantic annotation of emergency event cases. Paper presented at the First International 
Conference on Semantics, Knowledge and Grid, 2005, China. 
Wickens C.D. (1992) Engineering psychology and human performance. 2nd edn. Harper Collins, 
London. 
Wickens, C. D. (2008). Situation Awareness: Review of Mica Endsley?s 1995 Articles on 
SA theory and measurement. Human factors, 50 (3), 397-403. 
 Wilson, J. P. (2009). Policy actions of texas Gulf coast cities to mitigate hurricane damage: 
Perspectives of city officials. Applied Research Projects. Texas State University. 
XE ? Extreme Events (2002). Extreme events decision making workshop. Centre for 
policy research. Rockefeller College of Public Affairs and Policy. University at 
Albany. Decision Risk and Management Science Program. National Science 
Foundation. Available at: 
266 
 
http://www.albany.edu/cpr/xedm/XEDecisionMakingReport.pdf. Last 
accessed 6 February 2009. 
Yang, L., Prasanna, R., & King, M. (2009). On-site information systems design for 
emergency first responders. Journal of Information Technology Theory and Application, 
10(1), 5-27.   
Ye, J., Coyle, L., Dobson, S., Nixon, P. (2007). Ontology-based models in pervasive 
computing. The Knowledge Engineering Review, 22:4, 315?347. 
Yeh, T. H., & Lo, J. G. (2004) A case study of knowledge-based expert system for 
emergency response of chemical spill and fire. Environ. Inform. Arc. 2, 743-755. 
Yoon, S. W., Velasquez, J. D., Partridge, B. K., & Nof, S. Y. (2008). Transportation 
security decision support system for emergency response: A training 
prototype. Decis. Support Syst. 46, 1 (Dec. 2008), 139-148. 
Zhuge, H., Xing, Y., & Shi, P. (2008). Resource space model, OWL and database: 
Mapping and integration, ACM Transactions on Internet Technology, 8/4.  
Zmud, R. (1997) Editor?s comments, MIS Quarterly 21 (2), 21-22. 
Zsambok, C. E., & Klein, G. A. (1997). Naturalistic decision making. Mahwah, N.J.: 
Erlbaum. 
 
 
 
267 
 
APPENDIX A 
Research Information Sheet 
 
268 
 
 
 
 
 
269 
 
APPENDIX B 
Participation Consent Form 
 
270 
 
 
APPENDIX C 
Low Risk Research Approvals 
 
271 
 
 
 
 
 
 
 
272 
 
APPENDIX D 
General Interview questions 
 
 
 
273 
 
 
APPENDIX E 
Templates Used for Information Requirements Gathering  
 
 
274 
 
 
 
 
 
 
275 
 
 
 
 
 
 
276 
 
 
 
 
 
 
 
277 
 
 
 
 
 
 
 
278 
 
APPENDIX F 
Screenshots of the Ontologies Developed 
 
 
 
 
Concepts of SAVER ontology as seen in Prot?g? editor (Part 1) 
279 
 
 
Concepts of SAVER ontology as seen in Prot?g? editor (Part 2) 
280 
 
 
Hierarchy of asserted SAVER ontology as seen in Prot?g? editor 
281 
 
 
Hierarchy of inferred SAVER ontology as seen in Prot?g? editor 
282 
 
 
 
 
 
 
 
 
 
 
 
Partial SAVER ontology showing relationships between concepts  
283 
 
APPENDIX G 
Simulation Experiment Questions 
Part A: 
 AN EARTHQUAKE HAS OCCURRED WITH THESE PRELIMINARY PARAMETERS 
 ORIGIN TIME -  1334Z 01 APR 
 COORDINATES -  15.0 WEST  115.0 NORTH  
 LOCATION    -  MEXICO, NORTH PACIFIC OCEAN 
 MAGNITUDE   -  8.1 
 Depth       -  28 KM     
 
EVALUATION 
IT IS NOT KNOWN THAT A TSUNAMI WAS GENERATED. THIS WARNING IS BASED 
ONLY ON THE EARTHQUAKE EVALUATION. AN EARTHQUAKE OF THIS SIZE HAS 
THE POTENTIAL TO GENERATE A DESTRUCTIVE TSUNAMI THAT CAN STRIKE 
COASTLINES NEAR THE EPICENTER WITHIN MINUTES AND MORE DISTANT 
COASTLINES WITHIN HOURS. AUTHORITIES SHOULD TAKE APPROPRIATE ACTION 
IN RESPONSE TO THIS POSSIBILITY. THIS CENTER WILL MONITOR SEA LEVEL 
DATA FROM GAUGES NEAR THE EARTHQUAKE TO DETERMINE IF A TSUNAMI WAS 
GENERATED AND ESTIMATE THE SEVERITY OF THE THREAT.                               
 
1. What was the magnitude of earthquake? 
_________________ 
 
2. What was the depth of earthquake source? 
_________________ 
 
3. What was the location of the earthquake source? 
_________________ 
 
4. How well do you believe that you remembered the situation cues? 
o Perfectly 
o Sufficiently 
o Not enough 
o Not at all 
 
284 
 
5. According to your interpretation, earthquake magnitude is ________________ 
to generate a tsunami? 
o Very high 
o High enough 
o Not high enough 
o Not high at all    
 
6. According to your understanding of source depth and location, how likely it can 
cause a tsunami?  
o Highly likely  
o Likely 
o Unlikely 
o Highly unlikely  
 
7. According to your understanding of situation how likely it is that tsunami will be 
generated?  
o Highly likely 
o Likely 
o Unlikely 
o Highly unlikely  
 
8. What information / cues you perceived to understand the tsunami generation?  
Please list the cues you remember: __________________________________  
 
9. What do you require to understand / interpret the perceived cues? Or what have 
you used to understand or interpret the perceived information? Put number in 
front of options to rank.  
Experience _____ 
SAVER?s suggestion ______ 
Communication with partner ______  
Other ________________ _____ 
 
10. Once you have the understanding of information cues, what information you 
needed to predict the tsunami generation?  
Please list all sources: ___________________________________________ 
 
285 
 
11. According to your belief what informational cues your team member should 
perceive to understand tsunami generation? 
Please list the cues you remember: __________________________________  
 
12. What do you think your team member requires for understanding / interpreting 
the perceived cues? Or what have your team member used to understand or 
interpret the perceived cues? Put number in front of options to rank. ASSA2 
Experience _____ 
SAVER?s suggestion ______ 
Communication with partner ______  
Other ________________ _____ 
 
13. Once your team member has the understanding of information cues, what 
information he / she needed to predict the tsunami generation?  
Please list all sources: ___________________________________________ 
 
14. What do you think your team member answered about magnitude, depth and 
location of earthquake?  
Magnitude __________ 
Depth of source ____________ 
Location _____________ 
15. How well do you believe that your team member remembered the situational 
cues? 
o Perfectly 
o Sufficiently 
o Not enough 
o Not at all 
 
16. How confident you feel about your answer for the above question? 
o Highly 
o Somewhat 
o A little  
o Not much 
o Not enough 
o Not at all 
  
17. What do you believe that B will rate magnitude as: 
286 
 
o Low 
o High 
o Moderate 
o Don?t know  
 
18. According to your belief what your team member understands from source 
depth and location about the likelihood of tsunami generation? 
o Highly likely 
o Likely 
o Unlikely 
o Highly unlikely  
 
19. According to your belief what is your team member?s prediction about 
likelihood of tsunami generation?  
o Highly likely  
o Likely 
o Unlikely 
o Highly unlikely  
 
20. How well do you believe that your team member believe how well you 
remembered the situational cues? 
o Perfectly 
o Sufficiently  
o Not enough 
o Not at all 
 
21. How confident you feel about your answer for the above question? 
o Highly 
o Somewhat 
o A little  
o Not much 
o Not enough 
o Not at all 
 
22. Do you believe that your team member marked the same options as you did? 
o Highly agree 
287 
 
o Agree 
o Not agree 
o Don?t agree at all  
 
23. How confident you feel about your answer for the above question? 
o Highly 
o Somewhat 
o A little 
o Not much 
o Not enough 
o Not at all 
 
24. Do you believe that your team member believe that you marked the same 
options as he / she did? 
o Highly agree 
o Agree 
o Not agree 
o Don?t agree at all  
 
25. How confident you feel about your answer for the above question? 
o Highly 
o Somewhat 
o A little 
o Not much 
o Not enough 
o Not at all 
 
 
 
 
 
 
 
 
288 
 
Part B: 
DART sensors in the area provide the following wave information.  
MEASUREMENTS OR REPORTS OF TSUNAMI WAVE ACTIVITY ISSUED AT T5 2305Z 
01 APR 
 
GAUGE LOCATION                LAT   LON    TIME        AMPL         
PER 
 -------------------         ----- ------  -----  ---------------  -
---- 
240NM Manzanillo, Mexico     15.0N 110.0W  0244Z   1.31M =  4.3FT  
14MIN 
360NM Acapulco, Mexico       12.0N 100.0W  0124Z   1.58M =  5.2FT  
22MIN 
710NM Panama City, Panama    05.0N 090.0W  0135Z   1.88M =  6.2FT  
12MIN 
400NM Southeast of Tonga     08.4S 125.0W  0040Z   1.09M =  3.6FT  
40MIN 
640NM Auckland, New Zealand  33.0S 173.0W  2308Z   1.40M =  4.6FT  
62MIN 
 
MEASUREMENTS OR REPORTS OF TSUNAMI WAVE ACTIVITY ISSUED AT T4 2135Z 
01 APR 
 
GAUGE LOCATION                LAT   LON    TIME        AMPL         
PER 
 -------------------         ----- ------  -----  ---------------  -
---- 
240NM Manzanillo, Mexico     15.0N 110.0W  0244Z   1.37M =  4.5FT  
14MIN 
360NM Acapulco, Mexico       12.0N 100.0W  0124Z   1.61M =  5.3FT  
22MIN 
710NM Panama City, Panama    05.0N 090.0W  0135Z   1.92M =  6.3FT  
12MIN 
400NM Southeast of Tonga     08.4S 125.0W  0040Z   0.97M =  3.2FT  
40MIN 
640NM Auckland, New Zealand  33.0S 173.0W  2308Z   0.06M =  0.2FT  
62MIN 
 
 
MEASUREMENTS OR REPORTS OF TSUNAMI WAVE ACTIVITY ISSUED AT T3 1855Z 
01 APR 
 
 GAUGE LOCATION                LAT   LON    TIME        AMPL         
PER 
 -------------------         ----- ------  -----  ---------------  -
---- 
240NM Manzanillo, Mexico     15.0N 110.0W  0244Z   1.34M =  4.4FT  
14MIN 
360NM Acapulco, Mexico       12.0N 100.0W  0124Z   1.58M =  5.2FT  
22MIN 
710NM Panama City, Panama    05.0N 090.0W  0135Z   1.95M =  6.4FT  
12MIN 
400NM Southeast of Tonga     08.4S 125.0W  0040Z   0.06M =  0.2FT  
40MIN 
640NM Auckland, New Zealand  33.0S 173.0W  2308Z   0.09M =  0.3FT  
62MIN 
 
289 
 
MEASUREMENTS OR REPORTS OF TSUNAMI WAVE ACTIVITY ISSUED AT T2 1627Z 
01 APR 
 
GAUGE LOCATION                LAT   LON    TIME        AMPL         
PER 
 -------------------         ----- ------  -----  ---------------  -
---- 
240NM Manzanillo, Mexico     15.0N 110.0W  0244Z   1.24M =  4.1FT  
14MIN 
360NM Acapulco, Mexico       12.0N 100.0W  0124Z   1.55M =  5.1FT  
22MIN 
710NM Panama City, Panama    05.0N 090.0W  0135Z   0.03M =  0.1FT  
12MIN 
400NM Southeast of Tonga     08.4S 125.0W  0040Z   0.09M =  0.3FT  
40MIN 
640NM Auckland, New Zealand  33.0S 173.0W  2308Z   0.03M =  0.1FT  
62MIN 
 
MEASUREMENTS OR REPORTS OF TSUNAMI WAVE ACTIVITY ISSUED AT T1 1346Z 
01 APR 
 
GAUGE LOCATION                LAT   LON    TIME        AMPL         
PER 
 -------------------         ----- ------  -----  ---------------  -
---- 
240NM Manzanillo, Mexico     15.0N 110.0W  1340Z   1.28M =  4.2FT  
14MIN 
360NM Acapulco, Mexico       12.0N 100.0W  0124Z   0.09M =  0.3FT  
22MIN 
710NM Panama City, Panama    05.0N 090.0W  0135Z   0.06M =  0.2FT  
12MIN 
400NM Southeast of Tonga     08.4S 125.0W  0040Z   0.09M =  0.2FT  
40MIN 
640NM Auckland, New Zealand  33.0S 173.0W  2308Z   0.20M =  0.4FT  
62MIN 
 
 LAT - LATITUDE (N=NORTH, S=SOUTH) 
 LON - LONGITUDE (E=EAST, W=WEST) 
 TIME - TIME OF THE MEASUREMENT (Z = UTC = GREENWICH TIME) 
 AMPL - TSUNAMI AMPLITUDE MEASURED RELATIVE TO NORMAL SEA LEVEL. 
        IT IS ...NOT... CREST-TO-TROUGH WAVE HEIGHT. 
        IT IS ...NOT... CREST-TO-TROUGH WAVE HEIGHT. 
        VALUES ARE GIVEN IN BOTH METERS (M) AND FEET (FT). 
 PER - PERIOD OF TIME IN MINUTES (MIN) FROM ONE WAVE TO THE NEXT. 
 
 
290 
 
EVALUATION 
SEA LEVEL READINGS INDICATE A TSUNAMI WAS GENERATED. IT MAY HAVE 
BEEN DESTRUCTIVE ALONG COASTS NEAR THE EARTHQUAKE EPICENTER AND 
COULD ALSO BE A THREAT TO MORE DISTANT COASTS. AUTHORITIES SHOULD 
TAKE APPROPRIATE ACTION IN RESPONSE TO THIS POSSIBILITY. THIS CENTER 
WILL CONTINUE TO MONITOR SEA LEVEL DATA TO DETERMINE THE EXTENT AND 
SEVERITY OF THE THREAT.                                   
FOR ALL AREAS - WHEN NO MAJOR WAVES ARE OBSERVED FOR TWO HOURS AFTER 
THE ESTIMATED TIME OF ARRIVAL OR DAMAGING WAVES HAVE NOT OCCURRED 
FOR AT LEAST TWO HOURS THEN LOCAL AUTHORITIES CAN ASSUME THE THREAT 
IS PASSED. DANGER TO BOATS AND COASTAL STRUCTURES CAN CONTINUE FOR 
SEVERAL HOURS DUE TO RAPID CURRENTS. AS LOCAL CONDITIONS CAN CAUSE A 
WIDE VARIATION IN TSUNAMI WAVE ACTION THE ALL CLEAR DETERMINATION 
MUST BE MADE BY LOCAL AUTHORITIES.           
ESTIMATED INITIAL TSUNAMI WAVE ARRIVAL TIMES. ACTUAL ARRIVAL TIMES 
MAY DIFFER AND THE INITIAL WAVE MAY NOT BE THE LARGEST. THE TIME 
BETWEEN SUCCESSIVE TSUNAMI WAVES CAN BE FIVE MINUTES TO ONE HOUR. 
ESTIMATED INITIAL TSUNAMI WAVE ARRIVAL TIMES ISSUED AT 1627Z 01 APR 
 
LOCATION                            COORDINATES     ARRIVAL TIME 
 --------------------------------    ------------    ------------ 
 NEW ZEALAND      NORTH CAPE         34.4S 173.3E    0138Z 02 APR 
                  EAST CAPE          37.5S 178.5E    0214Z 02 APR 
                  AUCKLAND(W)        37.1S 174.2E    0238Z 02 APR 
                  GISBORNE           38.7S 178.0E    0247Z 02 APR 
                  MILFORD SOUND      44.5S 167.8E    0249Z 02 APR 
                  NEW PLYMOUTH       39.1S 174.1E    0310Z 02 APR 
                  NAPIER             39.5S 176.9E    0316Z 02 APR 
                  WESTPORT           41.8S 171.2E    0332Z 02 APR 
                  AUCKLAND(E)        36.7S 175.0E    0332Z 02 APR 
                  WELLINGTON         41.5S 174.8E    0333Z 02 APR 
                  BLUFF              46.6S 168.3E    0351Z 02 APR 
                  NELSON             41.3S 173.3E    0426Z 02 APR 
                  LYTTELTON          43.6S 172.7E    0439Z 02 APR 
                  DUNEDIN            45.9S 170.5E    0506Z 02 APR 
 
 
ESTIMATED INITIAL TSUNAMI WAVE ARRIVAL TIMES ISSUED AT 1346Z 01 APR 
 
 LOCATION                            COORDINATES     ARRIVAL TIME 
 --------------------------------    ------------    ------------ 
 SOLOMON IS.      MUNDA               8.4S 157.2E    2039Z 01 APR 
                  FALAMAE             7.4S 155.6E    2103Z 01 APR 
                  PANGGOE             6.9S 157.2E    2120Z 01 APR 
                  HONIARA             9.3S 160.0E    2121Z 01 APR 
                  GHATERE             7.8S 159.2E    2122Z 01 APR 
                  AUKI                8.8S 160.6E    2134Z 01 APR 
291 
 
                  KIRAKIRA           10.4S 161.9E    2140Z 01 APR 
 PAPUA NEW GUINE  AMUN                6.0S 154.7E    2124Z 01 APR 
                  KIETA               6.1S 155.6E    2133Z 01 APR 
                  RABAUL              4.2S 152.3E    2145Z 01 APR 
                  LAE                 6.8S 147.0E    2218Z 01 APR 
                  KAVIENG             2.5S 150.7E    2223Z 01 APR 
                  MADANG              5.2S 146.0E    2241Z 01 APR 
                  PORT MORESBY        9.5S 147.0E    2254Z 01 APR 
                  MANUS IS.           2.0S 147.5E    2259Z 01 APR 
                  WEWAK               3.5S 143.6E    2325Z 01 APR 
                  VANIMO              2.6S 141.3E    2350Z 01 APR 
 VANUATU          ESPERITU SANTO     15.1S 167.3E    2236Z 01 APR 
                  ANATOM IS.         20.2S 169.9E    2322Z 01 APR 
 NEW CALEDONIA    NOUMEA             22.3S 166.5E    2338Z 01 APR 
 AUSTRALIA        CAIRNS             16.7S 145.8E    2349Z 01 APR 
                  BRISBANE           27.2S 153.3E    0033Z 02 APR 
                  SYDNEY             33.9S 151.4E    0114Z 02 APR 
                  GLADSTONE          23.8S 151.4E    0139Z 02 APR 
                  MACKAY             21.1S 149.3E    0144Z 02 APR 
                  HOBART             43.3S 147.6E    0245Z 02 APR 
 TUVALU           FUNAFUTI IS.        7.9S 178.5E    2359Z 01 APR 
 KIRIBATI         TARAWA IS.          1.5N 173.0E    0007Z 02 APR 
                  KANTON IS.          2.8S 171.7W    0121Z 02 APR 
                  CHRISTMAS IS.       2.0N 157.5W    0325Z 02 APR 
                  MALDEN IS.          3.9S 154.9W    0336Z 02 APR 
                  FLINT IS.          11.4S 151.8W    0408Z 02 APR 
 FIJI             SUVA               18.1S 178.4E    0038Z 02 APR 
 KERMADEC IS      RAOUL IS.          29.2S 177.9W    0131Z 02 APR 
 NEW ZEALAND      NORTH CAPE         34.4S 173.3E    0138Z 02 APR 
                  EAST CAPE          37.5S 178.5E    0214Z 02 APR 
                  AUCKLAND(W)        37.1S 174.2E    0238Z 02 APR 
                  GISBORNE           38.7S 178.0E    0247Z 02 APR 
                  MILFORD SOUND      44.5S 167.8E    0249Z 02 APR 
                  NEW PLYMOUTH       39.1S 174.1E    0310Z 02 APR 
                  NAPIER             39.5S 176.9E    0316Z 02 APR 
                  WESTPORT           41.8S 171.2E    0332Z 02 APR 
                  AUCKLAND (E)       36.7S 175.0E    0332Z 02 APR 
                  WELLINGTON         41.5S 174.8E    0333Z 02 APR 
                  BLUFF              46.6S 168.3E    0351Z 02 APR 
                  NELSON             41.3S 173.3E    0426Z 02 APR 
                  LYTTELTON          43.6S 172.7E    0439Z 02 APR 
                  DUNEDIN            45.9S 170.5E    0506Z 02 APR 
 
 
1. What was wave height per _______ minute at sensor located at SAIPAN USA?  
________________________ 
 
2. What was wave height per ________ at sensor located at VANAUTA? 
________________________ 
 
3. How well do you believe that you remembered the situation cues? 
o Perfectly 
o Sufficiently 
o Not enough 
o Not at all 
292 
 
 
4. According to your interpretation of waves, wave height is ________________ 
to indicate the tsunami generation.  
o Very high  
o High enough  
o Not high enough 
o Not high at all   
 
5. According to your interpretation of wave heights, how likely is it that tsunami is 
generated? 
o Highly likely 
o Likely 
o Unlikely 
o Highly unlikely  
  
6. According to your understanding of wave travel direction, which areas are likely 
to be inundated? 
List as many you can remember:  _______________________________ 
 
7. According to your understanding of wave speed what is predicted arrival time of 
wave at Auckland coasts? 
Time: _____________ OR Hours _____________ 
 
8. What information / cues you should perceive to understand tsunami generation 
from wave information?  
Please list the cues you remember: _________________________________  
 
9. What do you require to understand from wave information that tsunami has 
been generated? Or what have you used to understand that tsunami has been 
generated? Put number in front of options to rank.  
Experience _____ 
SAVER?s suggestion ______ 
Communication with partner ______  
Other ________________ _____ 
 
10. What do you require to understand / interpret the speed of waves? Or what 
have you used to calculate the speed of waves? Put number in front of options 
to rank.  
293 
 
Experience _____ 
SAVER?s suggestion ______ 
Communication with partner ______  
Other ________________ _____ 
 
11. Once you have the understanding of information cues, what information you 
needed to predict the arrival time of tsunami wave on the coasts?  
Please list all sources: ___________________________________________ 
 
12. Once you have the understanding of information cues, what information you 
needed to identify the areas under risk of inundation?  
Please list all sources: ___________________________________________ 
 
13. According to your belief what informational cues your team member should 
perceive to understand tsunami generation?  
Please list the cues you remember: __________________________________  
 
14. What your team member requires to understand from wave information that 
tsunami has been generated? Or what have he / she used to understand that 
tsunami has been generated? Put number in front of options to rank.  
Experience _____ 
SAVER?s suggestion ______ 
Communication with partner ______  
Other ________________ _____ 
 
15. Once your team member has the understanding of information cues, what 
information he / she need to predict the arrival time of tsunami wave on the 
coasts?  
Please list all sources: ___________________________________________ 
 
16. What do you think your team member has answered about wave height at 
VANAUTA? 
Wave height: _____________ 
 
17. How well do you believe that your team member remembered the situational 
cues? 
o Perfectly 
o Sufficiently 
294 
 
o Not enough 
o Not at all 
 
18. How confident you feel about your answer for the above question? 
o Highly 
o Somewhat 
o A little 
o Not much 
o Not enough 
o Not at all 
  
19. What do you believe that B has rated wave height as ________________ to 
indicate the tsunami generation?  
o Very High 
o High enough 
o Not high enough 
o Not high at all   
 
20. According to your belief what your team member understands from wave 
information about the likelihood that tsunami is generated?  
o Highly likely 
o Likely 
o Unlikely 
o Highly unlikely  
 
21. According to your belief what is your team member?s prediction about wave 
arrival time at Auckland Coasts?  
Time: _____________ OR Hours _____________ 
 
22. How well do you believe that your team member believe how well you 
remembered the situational cues? 
o Perfectly 
o Sufficiently 
o Not enough 
o Not at all 
 
295 
 
23. How confident you feel about your answer for the above question? 
o Highly  
o Somewhat  
o A little 
o Not much 
o Not enough 
o Not at all 
 
24. Do you believe that your team member marked the same options as you did? 
o Highly agree 
o Agree 
o Not agree 
o Don?t agree at all  
 
25. How confident you feel about your answer for the above question? 
o Highly 
o Somewhat 
o A little 
o Not much 
o Not enough 
o Not at all 
 
26. Do you believe that your team member believe that you marked the same 
options as he / she did? 
o Highly agree 
o Agree 
o Not agree 
o Don?t agree at all  
27. How confident you feel about your answer for the above question? 
o Highly  
o Somewhat 
o A little 
o Not much 
o Not enough 
o Not at all