| Current practices to defend against cyber attacks are typically reactive yet passive. Recent research work has been proposed to proactively predict hacker's target entities in the early stage of the attack. With prediction, there comes false alarms and missed attacks. Very little has been reported on how to evaluate a threat assessment algorithm, especially for cyber security. Because of the variety and the constantly changing nature of hacker behavior and network vulnerabilities, a cyber threat assessment algorithm is, perhaps more susceptible that for other application domains. This work sets forth the issues on evaluating cyber threat assessment algorithms, and discusses the validity of various statistical measures. Simulation examples are provided to illustrate the pros and cons of using different metrics under various cyber attack scenarios. Our results show that commonly used false positives and false negatives are necessary but not sufficient to evaluate cyber threat assessment. |
| Dr. Shanchieh Jay Yang received his M.S. and Ph.D. from the University of Texas at Austin in Electrical and Computer Engineering in 2001. After working as consultant / research scientist for NetQoS, Fujitsu Laboratories of America, and General Bandwidth, Dr. Yang joined the Computer Engineering department at Rochester Institute of Technology as an assistant professor in 2002. His current research focuses on autonomous sensing and information fusion networks. Specific projects include high level information fusion for cyber security and collaborative robot and camera surveillance. In the past, he has also worked on energy efficient wireless ad hoc sensor networks, Internet user impatience modeling, dynamic network resource allocation, and micro/macro mobility management for IP networks. Dr. Yang has served on the technical committee and reviewers for book chapters as well as conference and journal publications, such as IEEE Infocom, IEEE Globecom, IEEE MASS, and IEEE/ACM Transaction on Networking. Dr. Yang was the Co-chair for the IEEE Joint Communications and Aerospace Chapter in Rochester NY in 2005 and has helped form the Annual IEEE Upstate NY Workshop on Communications and Networking. |