Costing Out an AQP Development Effort

As air carriers begin assessing the utility of converting their training programs to an AQP format, the first question usually asked by upper management is, “How much is this going to cost us?”  This paper gives some thoughts on how to proceed in “costing” out an air carrier’s AQP.

The term “cost” usually incorporates two major themes: time and money – how long will it take to implement the program and how much money is going to be spent to get it up and running?  The driving factor on answering those questions is to adequately define the scope of your program:

 How many fleets need to be converted?

 How many curricula will be affected?

 Are you going to implement Single Visit Training (SVT) first, or jump right into a full-blown AQP?

 How are you going to develop and maintain your Program Audit Database (PADB) and your Performance/Proficiency Database (PPDB)?

 A word processing program (e.g., MS Word, Notepad, etc.)?

 A spreadsheet program (LOTUS, MS Excel, etc.)?

 A database program (MS Access, MSSQL Server, Oracle, Paradox, FoxPro, etc.)?

 How are you going to analyze and use the information in your PPDB?

 How are you going to disseminate information on your AQP – the Job Task List (JTL), the Qualification Standards (QSD), the Curriculum Outlines, your data analysis reports, etc?

 Printed documents?

 Electronic documents?

 Collaboration space (e.g., company intranet, SharePoint portal, etc.)?

 Reporting Services, Excel Services, etc.?

It is important that the air carrier make a conscientious decision on their program’s scope – are we trying to do the minimum necessary to meet the Advisory Circular requirements in order to gain the advantages of being an AQP carrier – i.e., develop the required documents, collect the required data, and make the required reports to the FAA; OR, are we trying to utilize the full flexibility of AQP to develop a training paradigm within our company to provide the safest, best trained crew members and support personnel that we possibly can?  For the sake of this paper, let us assume that the scope choice is the latter – a meaningful AQP that includes data quality assurance and feedback loops.

With scope defined, we can now begin the task of putting our “team” together.  AC 120-54a provides some insight into this task in its discussion of the Extended Review Team, or ERT (paragraph 1-5a3).  My experience with developing AQPs for several carriers leads me to define the “team” as consisting of:

 AQP Manager – usually a highly motivated instructor with expertise on training department operations and procedures who has been qualified on a majority of the carrier’s aircraft.  This individual initially concentrates on getting the program up and running within one fleet but eventually gravitates to developing AQPs across all fleets and crew positions.  As the majority of the carrier’s training programs migrate from traditional Part 121 training to AQP, this position morphs into, or responsibility transfers to, the carrier’s Director of Training.  However, make no mistake that, at the outset, this position must be filled by a dedicated individual whose sole function is to get your AQP up and running and is not working, part-time, on other Part 121 programs.

 Subject Matter Experts (SMEs) – these individuals are usually instructors qualified on the aircraft for which you are currently developing AQPs.  Their job is to assist with development and review of such products as the Job Task List, Qualification Standards and Curriculum Outlines.  I would recommend two dedicated individuals so that product reviews are not held up due to a SME flying line trips, providing instruction, being on vacation, etc.  Both are required at startup and need to remain on the team until the final Curriculum Outline is produced.  At that point, when all AQP efforts within a fleet are finally into a maintenance/review status, you could drop the requirement to one SME.

 Instructional Systems Designer (ISD)/Curriculum Specialist – an individual with a thorough background in objective-based training development, creation of job task lists and standards documents, training media selection, and curriculum development.  This individual needs to remain on the team until the last AQP Curriculum Outline is produced.

 Database Developer – an individual with a thorough background in your underlying PADB and PPDB databases (MS SQL, Oracle, Access, etc.).  This individual is responsible for designing and developing your required databases, ensuring data quality, and maintaining the associated databases.  Unfortunately, there is no one product available on the market today that seamlessly integrates a PADB with a PPDB while simultaneously providing a data analysis and reporting capability and generates the FAA’s required reporting information (see AC 120-54a, Chapter 8).  There are, however, various products that can be “cobbled together” to provide some of this capability (e.g., Battelle’s AQP Model I and/or Proteus, and the FAA’s Data Analysis and Reporting Tool (DART) - see AC 120-54a, Figure 1-1).   As your databases stabilize and require less and less modification, this individual’s need as a dedicated team member diminishes while that of the Business Analyst (see below) increases.

 Business Intelligence (BI) Analyst – an individual with a thorough background in developing and delivering analytical reports on the data contained in the PPDB.  While many companies tend to pass this requirement off to their Information Technology (IT) department (if your company is fortunate to have such a department), you will find yourselves at the mercy of IT’s development/maintenance schedules as your analysis requirements change and grow over time.  While not immediately required on the team, this individual’s need increases as the underlying PPDB stabilizes in its design and grows in size over time.

 Software Developer – In conjunction with the Database Developer and BI Analyst, this individual’s responsibility is develop methods for exposing AQP information to the requisite stakeholder’s during the program’s development and throughout its subsequent operation.  This includes not only methodologies for making AQP documentation available to all but also PPDB data collection and dissemination of analysis results. 

 Data Entry Clerk – Given AQP’s dependence on databases and data collection, this individual is the primary person responsible for data entry into both the PADB and the PPDB.  Depending upon the size of the carrier’s operations or its data collection methodology, this could be a part-time or full-time position. 

Depending upon their backgrounds, the Database Developer, BI Analyst, and Software Developer positions could be combined into one individual.  It is important to recognize that while most companies might possess an IT department that could provide these services, it is imperative that individuals be dedicated in those positions to the AQP project.  Two previous carriers that I have worked with had IT departments that were unable to meet the AQP development needs.  In the case of one carrier, the IT department worked for three years transitioning a functioning MS Access SVT data collection and management application over to an AQP application using MS SQL Server.  In the end, the IT department returned the development project back to the training department due to their (IT’s) inability to dedicate individuals to the transition project.

On the issue of “how long will it take?”, it is, quite simply, forever.  There is no middle ground with AQP.  Once you decide to convert to AQP, you do so “lock, stock, and barrel!”  Carriers must recognize that the end game with AQP is a complete conversion of all existing and future traditional Part 121 training programs to AQP.  However, by breaking AQP development into various sub-components, we can begin to estimate some development times.

There are several deliverables to the FAA during each of the five phases of development.  In regards to time, Phase II, Curriculum Development, is the most demanding.  With no prior AQP development experience, you can expect to spend some 12-18 months developing your first Job Task List/Analysis (JTL/JTA) and Qualification Standards (QSD), especially without having an existing JTL/QSD from which to work.  One of the most critical factors affecting development time is the choice of what level of detail to choose for your JTL – too high and you lack the fidelity with which to collect data to make meaningful changes to your program, too low and you’ll get bogged down in the details. 

Likewise, the level of detail of your objective hierarchy can contribute to several rewrites of your QSD.  Again, too much detail will result in a Continuing Qualification (QC) program that quickly becomes unmanageable while too high a level of detail results in a QC lacking in any “meat.”  After your first JTL and QSD submissions, subsequent fleet JTLs and QSDs can be produced in roughly half the time the initial submission required.

Once your JTL and QSD for your first fleet are submitted to the FAA, it’s time to begin developing your Curriculum Outlines.  Your first major decision here is whether to target CQ, Indoctrination (Indoc), or Qualification (Qual).  Most carriers choose to develop CQ first in order to eliminate the semi-annual Captain’s Proficiency Check requirement under traditional Part 121 training and to begin getting some “payback” on their decision to enter AQP training.  Properly constructed, you will probably see a much improved Recurrent Training program with CQ in which your aircrews will receive better and more line oriented training than under traditional Part 121 training (However, I have yet to see where a properly constructed CQ results in an overall lower cost training program than under traditional Part 121 training).  You can expect to spend between 4-8 months developing your first CQ program – including instructor training and small group tryouts.

If your carrier decides to go the Indoc/Qual route first, you can expect to spend 8-12 months developing your first fleet Qual Curriculum Outline (CO).  Subsequent COs will take approximately 2-4 months to develop as they are merely adaptations of the baseline Qual curriculum – but there’s lots of them to develop: Transitions, Requals, Upgrades, Downgrades, etc.

And, while you’re developing those first fleet Curriculum Outlines, don’t forget you need to do parallel development of Instructor/Evaluator (I/E) Indoc, Qual, and CQ Curriculum Outlines and Small Group Tryouts (not to mention the I/E JTL and QSD).

So, in terms of time, you can probably expect the first fleet’s CQ development to take roughly 16-26 months of development time from when you make the decision to go ahead until you’ve got an up and running CQ.  You should plan on 20-30 months if you’re initially going with a Qual Curriculum. 

Of course, these estimates will vary depending on whether or not you have personnel with AQP development experience; the point you are at when you enter AQP development – traditional Part 121 training or SVT; and how many personnel you can put on the team and their backgrounds.  However, the estimates contained herein come pretty close to the FAA’s requirement of 30 months for completing Phases II and III (see AC 120-54a, Figure 4-1).

Your whole team is engaged while trying to complete Development Phases I-III for just your first fleet’s initial curriculum.  Once that program gets into Phase IV, their efforts then switch to your next curriculum development for your fleet (a process that will take between 2-4 months for each curriculum) or introduction of AQP into another fleet (a process that will take between 10-18 months for each additional fleet).  Most carriers will stagger fleet introduction to AQP by 3-6 months in order to speed up the time it takes to transition all fleets over to AQP.

The AQP Manager and ISD person’s efforts early on in Phases I and II are directed towards getting the other required documents (the Application, Instructional Systems Development Methodology, and Implementation & Operations Plan) completed as well as getting the program off to a good start.  Your database and software developers are simultaneously working to get your PADB, PPDB, and information distribution systems up and running in time to support your first fleet’s entry into Phase IV, Initial Operations.  Once up and running, their efforts then move to database maintenance, quality assurance, and data analysis and reporting.

As new fleets enter into AQP development, you team size will increase by two SMEs per fleet and possibly a fleet AQP coordinator (could be one of the SMEs) to reduce the workload on the overall AQP Manager.  At some carriers with multiple fleets and thousands of pilots, Quality Assurance departments have sprung up that monitor not only AQP but all Voluntary Disclosure Programs that are coordinated through AFS-230.

So, back to the original question – “What’s this going to cost?”  Your biggest expenses will be personnel costs, additive to your existing training department’s overhead.  As has been seen, for the first year or so, you are basically running parallel training departments.  You could use your own carrier’s pay scales to determine the annual costs for the AQP Manager and SME positions.  You could talk to your IT department about database and software developer salaries.  You could also use a tool like Salary.com to estimate annual salaries within your locale for the BI Analyst, ISD/Curriculum Specialist, and Data Entry positions.

Your other major cost category is going to be for software – the tools you use to create and maintain your databases and the information systems used to distribute AQP documents and associated analytical analyses and reports.  Before blindly going out and selecting items, talk to your IT department or its equivalent.  What systems are you currently using?  Are you Windows or Unix-based?  What office productivity software do you use – is it Office, Symphony, or something else?  You will find that life will be much simpler using software that is compatible with your company’s existing or planned systems. 

Once you have an understanding of what your company uses, you can then look for complementary systems that might have a price advantage.  For example, do we really need MS SQL Server Enterprise edition or can we get by with MS SQL Server Express?   If we have a Windows Server, can we get by with the “free” Windows SharePoint Services or do we need to purchase Microsoft Office SharePoint Server?  You will find that taking the time to adequately address and resolve your software requirements will go far in avoiding program development restarts downstream.

I hope that this paper has given you some insights into the pitfalls associated with striking out on an AQP development effort.  While every project that I have been associated with has been faced with challenges, in the end we were able to meet them head-on, resolve them, and successfully proceed to develop a premier AQP that provided safe, effective, and exceptional training programs to our aircrews.