Why Evaluate Your Program
Part of annual policy review
If you don’t evaluate you will never improve
Continual review will help protect your budget
Start At The Outside and Move Your Way In
Awareness and Education is how most people in your org know the program
Threat Mapping maps the outside threats to your inside controls & tech
Communications is that final turn from the inside out
Measuring Awareness & Education
What do you think you do?
Mandatory CBLs
CyberCyberCyberStuff (Posters, Email, Swag)
Briefings and Classes
Phishing Awareness
$NOVEL_IDEA
How do you measure it?
How many people is it designed to engage?
How many people were actually engaged?
Not how many people took the awareness, how many people were ENGAGED?
How did they do? (CBL completions, % phished, reviews, etc)
Are you being honest with yourself?
If CBL_Completion = 15(clicks) then you may want to rethink that
0% phished is not a sign of a great security program...more likely a sign of a bad phishing program
If there is no way to allow for anonymous reviews of training/briefings/etc then you’re not likely to get fully honest reviews (Who wants to piss off security?)
Adjusting The Program
Don’t change the measurement...change the program
The key to long term success is consistently measuring the same thing over time
You may want to update goals (up or down) but be able to explain why especially if you are making the test easier
Don’t make drastic changes until Year 3 unless you have to make drastic changes
Big changes in delivery will skew the numbers in ways you likely will not like
Constant large turmoil is counter to most corporate cultures
Small changes take advantage of previous investments best
“Iterate small and grow larger” - doing too much too fast almost always ends is highly suboptimal results over time
Clearly failing components should be axed and replaced and not tweaked around the edges - especially if there’s a compliance or safety aspect
If this feels like “Wash, Rinse, Repeat” it’s because is it “Wash, Rinse, Repeat”