Internal Evaluation Programs: Headache or Helpful Tool?

I recently presented a webinar for the National Air Transportation Association (NATA) on common audit “gotchas” – issues that seem to pop up frequently when I’m conducting third-party audits or helping clients prepare for these audits. (If you missed the webinar, an archived version is available here.)

While conducting audits or helping clients prepare for an audit, I often discover inadequacies in the development and implementation of internal evaluation programs (IEPs). Part 135 operators are not required to have an IEP by regulation but many Part 135 operators now have one. A well-written and properly executed IEP can be a great asset in the Safety Assurance aspect of a Safety Management System. (You remember the four tenets of SMS, right? Safety Policy, Safety Risk Management, Safety Assurance, and Safety Promotion.)

I’d like to believe the charter operators I work with or audit have an IEP as a moving part of the SMS but I’m not naïve. Many Part 135 operators have an IEP to please the multitude of audit standards and often the quality of these “check a box” IEPs is … lacking.

Look at the past year’s IEP forms and answer these questions honestly:

Do you always “pass” with flying colors and no findings? EVER?

Do you have findings that are never addressed?

Do your operational control, flight following, and training IEP items all fit on a single page?

If you answered “yes” to any of these questions, you might have an inadequate IEP. I often see these issues come up with purchased, off-the-shelf IEPs. Some of the off-the-shelf IEPs are made up of questions that are so incredibly broad that only someone very well-trained in auditing and the subject matter itself could answer the question with any validity.

For example, your IEP includes this question: “Are operational control procedures defined and properly implemented?” You hand over the IEP form to the most senior person in your dispatch/flight following office and ask them to complete the checklist. Hopefully this person has been properly trained in operational control but most likely they’ve only been trained to their own tasks or responsibilities. How do they know if operational control procedures are defined and properly implemented? Operational control procedures cannot be verified with a single question unless the individual conducting the evaluation knows the many implications behind this seemingly simple question.

Not all canned IEPs are bad or inadequate. Often the deficiency is in the implementation. In the example above, the IEP question itself is pretty weak but lack of training of the individual conducting the evaluation also negatively impacts the IEP’s effectiveness. Are the individuals involved with your IEP trained as auditors? This training doesn’t need to be a formal certificate program from a university, but they should have at least basic training in the audit process and in your IEP.

The final concern I find with many IEPs is a lack of follow through. KNOWING you have a problem and choosing not to fix can actually be worse than not knowing in the first place. The FAA takes a dim view of operators that identify a regulatory problem but just cover it up instead of fixing it. If you ever have an accident, the plaintiff’s attorney might retire off an IEP checklist with a noted deficiency that was never addressed. (Please note I’m not saying to rig your IEP or stop doing an IEP so as to avoid producing “evidence” that could be used against you. At this point, SMS and all of its moving parts – including IEPs – are an industry best practice and are moving towards becoming an industry standard. It’s possible you could be found negligent for NOT fully implementing an SMS, but more on that another day.)

Remember, the IEP should be a TOOL. It should help you identify weak areas before the FAA finds them in an inspection, an auditor uncovers them during an audit, or the NTSB discovers them in the wreckage. It’s a TOOL. My dad always taught me to use the right tool for the project – no butter knives as screw drivers or blocks of wood as hammers. How do you develop and implement a tool that is right for your company?

Here are three “simple” steps to developing and implementing a useful IEP:

1. Write an IEP that is appropriate for your organization. Look at the third-party audit standards to which you hold yourself accountable or the FAA’s Advisory Circular on IEPs for suggestions and guidance (AC 120-59A). If you don’t conduct cargo-only operations, don’t have a bunch of cargo-only questions in your IEP. Marking a bunch items “Not Applicable” is inefficient and can lead to complacency.

2. Train the individuals who will be involved in your IEP. In fact, get them to help you develop the checklist. Asking for their advice and input will help with buy-in when it’s time to implement.

3. Implement with intention. I understand your company’s mission is to fly airplanes, not complete checklists, but the IEP should be a priority. Set reasonable due dates and stick to them. Develop a system to track progress on any findings, assign an individual to be responsible for each finding, and set due dates for completion and further evaluation. This requires a top-down commitment to the program. Middle managers and line personnel need to know following through with an IEP finding is as critical as following a pre-flight checklist, completing a maintenance work card, or conducting flight following. Don’t bother creating a program you won’t enforce.

Take the time to develop the right tool – the right IEP – for your operation, train the individuals involved in the program, and then treat it as the helpful resource it should be. Don’t check a box and move on.

After that moving soapbox, a little humor: I often see IEP checklists marked with different completion dates but… the SAME INK. The exact same ink. On fresh, white, clean pages. I challenge you to complete a tool control checklist without getting dirt on the paper. (I know some people ALWAYS use the same pen but there are other clues that can raise my suspicion.)

This post is the first in a series of audit “gotchas”. I’ll be talking about training program issues, flying managers, and other common audit concerns over the next several weeks, so stay tuned!

Click here to return to or visit McFarren Aviation Consulting’s website.

2 thoughts on “Internal Evaluation Programs: Headache or Helpful Tool?

  1. Pingback: Managing Flying Managers | McFarren Aviation Safety News

  2. Pingback: Crewmember Records: Low Hanging Fruit | McFarren Aviation Safety News

Have some thoughts on today's post? Leave me a comment! All comments will be moderated to avoid spam or inappropriate comments so don't worry if you don't see your comments right away!

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s