Intro: Welcome to the E6S-Methods podcast with Jacob and Aaron, your source for expert training, coaching, consulting, and leadership in Lean, Six Sigma, and continuous improvement methods. In this episode number 55, “Rule out the Ruler – Part 4A,” we introduce the standards and discuss our personal preferences in this first of two episodes on Attribute Agreement Analysis. Here we go.

***Rule out the Ruler Part 4- Attribute Agreement Analysis***

I Scenario: Morning Startup Routines

a. Spray coatings department, process centering process performed at the start of each shift

b. Each machine is given a “go” or “no go” for production based on the quality of a test button, based on quality of coverage and centeredness.

i. Ownership of this determination kept getting handed back and forth between shop personnel and engineers. Shop personnel did not want to be accountable for making a bad choice. Engineering wanted shop personnel to take ownership of the quality of production, and to learn how to make this decision for themselves.

c. An Attribute Agreement Analysis was performed among the Engineers, the so-called “experts” at making this “go/ no go” decision.

i. Engineers made the same decision only 50%-60% of the time, and only made the “correct” decision 20%-40% of the time.

ii. In all, out of 20 parts evaluated, the engineers failed to make ANY repeatable & correct decisions-- fully agree with themselves, each other, and the standard on any part.

II How to perform & interpret MSA data (The official way)

a. Select 30 parts from the process – 50/50 pass/fail

b. Select marginally good and bad samples

c. Select fully trained inspectors

d. Each inspector inspects the parts in random order twice, recording the results

e. Analyze the data, decide if adequate

i. AIAG rules for Discrete (Kappa)

1. coefficient to indicate agreement percentage against that of pure chance

a. -1 perfect disagreement, much worse than by chance alone

b. +1 perfect agreement, much better than by chance alone

2. Some varying rules of thumb:

f. Implement any improvements

g. Re-run Gauge Study

h. Document findings

III Tips & Experiences

a. Select 30 parts from the process – 50/50 pass/fail

i. Modify if not doing a “pass/fail,” “yes/no” binary assessment

1. Decisions on call, complaint or defect type classifications

ii. More parts helps shrink the confidence intervals. Aaron has done assessments with as little as 10 parts. Sometimes it’s very easy to find gaps in disagreement without a full-on blind study.

b. Select marginally good and bad samples

i. Be careful of biases. Much like a survey, inspectors may start giving you what they think you want to see. When selecting parts, biases can be imposed.

ii. Aaron often just chooses parts randomly then adding some clear known good and known bad parts, representing the process.

c. Select fully trained inspectors

i. Side note: Attribute agreement is a good measure to qualify a new inspector. Also a good ongoing recalibration exercise

d. Each inspector inspects the parts in random order twice, recording the results

i. Perhaps even a 3rd time if not enough samples and/or it’s easy to remember previous responses

e. Analyze the data

i. Beware the destructive forces of inspection

1. M&M visual quality fades each time you handle them

2. Cannot taste test the same M&M twice

3. Reclassifying the same call, but out of context of call-order can create a bias twist

4. Also consider sample prep is a factor in attribute agreement, not just assessing the final sample.

f. Implement any improvements

i. Retraining

ii. Updating standards & tools

Outro: Thanks for listening to episode 55 of the E6S-Methods Podcast. Stay tuned for episode number 56 where we highlight some of the dangers and biases created during the assessment process, finishing up our discussion of the Attribute Agreement Analysis. If you would like to be a guest on podcast, contact us through our website. Follow us on twitter @e6sindustries or join a discussion on LinkedIn. Subscribe to past and future episodes on iTunes or stream us live on-demand with Stitcher Radio. Find outlines and graphics for all shows and more at www.E6S-Methods.com. “Journey Through Success”