Fourth Annual DNA Grantees' Workshop

Wednesday, June 25, 2003

AFTERNOON SESSION

Question-and-Answer Session

MS. BASHINSKI: Thanks. Well, we have plenty of time for discussion, but I know a lot of people are going to want to catch their flights, too.

I just wanted to ask one question of both Cecelia and Roger. You alluded to expert systems. Cecelia said that you had an analyst who was a little recalcitrant. We're all kind of risk-averse or change-averse. Do you have any thoughts on how to deal with staff like that when it comes to both robotics and expert systems?

DR. CROUSE: Well, first of all, the person that was recalcitrant to change has a date with Eddy on Friday. She actually fell in love with him, so it will change. It will change their minds. Now they can't imagine not having a robotics system. It's there, it can go into your laboratory.

We're getting set up now to do automated quantitations as soon as we figure out exactly which one we want. So we're doing the Rube Goldberg add-on because we're not that big of a lab. But small labs can do this.

With regard to expert systems, last week I did a case that had five small pinhead stains, thus interpretation hell. You wonder what the reviewer's going to say, and then you're thinking, no, I can't wonder what the reviewer is going to say. I've just got to make these calls. And of course, there are no standards anywhere, so you don't have any—which you shouldn't have anyway. But regardless, it's very, very difficult.

I certainly understand expert systems in CODIS. I definitely understand that and how that would work. But some of these profiles are pretty fuzzy. I mean, you guys look at it in terms of RFUs (relative fluorescence units) and we look at it in terms of does it meet the definition of a band. I'm just going to have to see it work. We've got to get the gels out there, and we've got to get them working on a system.

I was interested yesterday when they were trying to make a difference between an expert system and quality assurance tool. So now there are two different issues and I've got to kind of backtrack and figure that one out.

DR. KAHN: I agree. Just a quick comment. When I first started working in the crime lab, I walked around it and I went into the firearms section. I asked Jimmy Carr, who's now the director of that lab, why the lab didn't use computers to compare. He really got excited. It was probably the dumbest thing he had ever heard. He said that computers are not identical, and it takes a human eye to make the necessary contortions and manipulations and mind-numbing transformations to see them and they're the same. And then there was DRUGFIRE. He was really excited about it, and he uses it. DRUGFIRE has since been replaced by a more sophisticated and expensive system that improved it, but it still requires human intervention. It ranks the candidates and so on.

I don't even care if there are limitations to an expert system that does calling. I just want some help. If the system can do some of them, then that's terrific. If it can do most of them, then that's better. And if it can do all of them, then we'll find lots of work for folks to do.

I don't think that it'll be possible to go to court without a human, and I think there are many other issues to contend with, for example, deciding along with the investigators what should be tested and arranging the samples onto these machines and seeing that the reports go where they're supposed to go. And then there is the whole area of the impact of our work through disposition; we don't do a very good job at finding out what happens to the cases.

I think there will be plenty of work to do. So from my perspective, I do believe that it's going to happen. Of course you've got to show me that it works, but I have no doubt that it will happen because there are so many of us who would benefit from it.

MS. BASHINSKI: Anybody else on the panel have a comment about expert systems?

MR. BAN: Let me back up a little bit to what Cecelia said about the robot. I pretty much have to concur. I have a large staff and it was a little bit of a challenge initially when we first started using the robot. We took a very strong approach. We did this little by little, but once you start using the robot, that's it, you're pretty much hooked. You realize that it takes about 2 minutes to load a sample, and in 55 minutes, you come back to pick up a tube with an extracted sample. It automates the quantitation step. Eventually it's going to automate the dilution and the amplification step.

So, I am in agreement with what Roger is saying: I don't think this is ever going to put people out of work. We've got way too much work. Maybe it will reduce our backlogs, but it's definitely not going to put people out of work.

As far as expert systems, I have been looking at expert systems for a couple of years. Cecelia and I have been working together on this. It's definitely something that is needed. As a member of SWGDAM (Scientific Working Group on DNA Analysis Methods), I recognize that SWGDAM is beginning to look at this a little more seriously.

A year ago, it was interesting, if you talked to people about expert systems, they hated the whole concept. We've tested one of the expert systems in my laboratory for databanking purposes, and initially it was a real tough thing to get started, and now the staff really likes the concept of being able to walk away, come back, and have your data prepared for you.

So it's definitely where we're going, whether people like it or not. It's a tough thing to change, especially for the people who have been doing this for many years. But I think once you've done it, you're not going to turn back.

MS. BASHINSKI: Thanks.

I have one other issue which is kind of related to all of this. It has to do with LIMS. Many people have one variety or another. As Roger said, you get something off the shelf and then you have to create something that works in your own lab. If it doesn't have the approach that you might want, then you have to expend a lot of energy.

When I was with the Department of Justice, we spent literally 5 years putting together a statewide LIMS which basically didn't make anybody happy. It just has too many things that we weren't able to refine or make perfect.

But it's my sense that if everybody is spending all this time on it, then there must be something that we can gain from all the work and energy and effort that people have put into these systems already.

I know that SWGDAM has a computer committee, but they don't necessarily work with something like LIMS. You're talking more about CODIS itself and the data, as I understand it. Is there a group that could look at the kind of product that people have in their labs and maybe serve as a clearinghouse, or can NIJ or the Forensic Resource Network or somebody like that leverage all of this wheel spinning that we've done?

DR. CROUSE: I know in our laboratory we were going to go online on January 1, 1996, and lo and behold, last June I started using it. So it did take a long time. I know the person who was in charge of LIMS for our technical service bureau went to an annual meeting in Arizona or someplace, and she just got back. It's important for anyone using these systems in the crime labs to go out there and talk about the systems, so they can be improved. I think it's come a long way. There's a lot of sharing going on, I think.

DR. KAHN: I can offer one small thing. If you have an opportunity, ASCLD (American Society of Crime Laboratory Directors) usually offers a workshop at the front end of their meeting. Cindy Shannon is helping to put this on. I don't know if Cindy is here, but she's the CODIS administrator who works with us and was instrumental in putting our LIMS together.

For this particular workshop, we have asked one person from each laboratory, who self-identified themselves as a successful owner of a LIMS product and who really love it, to get up and present how they were able to be successful. I think what they're going to all say is that they spent an awful lot of time in design, far more than a LIMS vendor would ever suggest you need. I think we've got seven systems or six, something like that.

You'll find out that the actual cost of your LIMS is a bunch more than the cost of the site licenses for the off-the-shelf, shrink-wrapped product. I don't know but it probably varies from site to site.

MR. SHELTON: I have a question for Jeff.

DR. FORMAN: Would you identify yourself.

MR. SHELTON: Harry Shelton.

I was just wondering if there have been any court challenges (inaudible).

MR. BAN: We routinely turn procedure manuals, case manuals, and whatever over for discovery. We have a lot of defense experts that like to visit Virginia, for some reason, but we have not had any challenges on it yet.

The way we look at it is this plate control that we've introduced is no different than your reagent blank and, ironically, we've had more situations where a reagent blank has come up with a result manually than a plate blank has using a robot. A lot of that is because of human error. The precision of the pipetting of the instrument is more precise than you as a human can do.

MS. BLOSSOM: I wanted to ask, you spoke about (inaudible).

MS. BASHINSKI: For the notes we need you the identify yourself.

MS. BLOSSOM: Mary Blossom.

MS. KIDD: Mary is asking a question asked by a lot of people. The question is how do you store the data, having to reproduce it for discovery. Normally, we just take the case file and run the whole thing through the color copier. We have color copiers at our disposal, so that helps a lot.

We also save the data to hard drives. We used to save it to what we call our G drive, which was kind of lab-shared all over, but we realized that that could be opening a huge door for discovery. So right now we're just saving it all on the serologists' hard drive. Since we just moved to a new laboratory, a lot of the material probably got downloaded to a CD.

But that's what we're doing. Nobody has asked us in discovery for all of the data off someone's hard drive. We do have an excellent legal counsel that I think would argue vehemently about turning over an entire hard drive, because it's not case specific and it has other information on it as well. But we haven't had any challenges so far.

So how do you know if something's been changed? If there is a change made, I will make it right to the worksheet and cross it out and initial it. So if somebody tries to reproduce it, good luck.

So far it has worked very well. As I said, we've gotten a lot of good feedback from defense experts saying how much easier our case notes are to review and how clear they are, and we've only had a few people come back and ask us questions like they used to.

MS. BASHINSKI: How do you handle that, Roger? You have a report that gets put together with pieces. The computer is putting together a report. Are you part paper and part electronic or what?

DR. KAHN: You can see what it's doing. The boilerplate language that goes in is in the wizard in front of you before it actually assembles into the report. And then of course it's all tech reviewed, so it's checked for accuracy before it goes out.

MS. BASHINSKI: Is it locked? Is it locked then after that?

DR. KAHN: The reports are all exported into Word, which are saved as Word files. If an error occurs, then you just make a new one by going back into the wizard and starting again. We ask people not to edit the Word files because the data behind the report are actually in the database maintained by the LIMS. They don't have to do that, but it would be better to be able to continually recreate the report and show that it can be done using the LIMS.

MS. BASHINSKI: So how do you know what the official report is, then?

DR. KAHN: The Word file is it. The Word file is the report.

MS. SHANNON: Hi. I'm Cindy Shannon with BCI.

After you go through and answer your questions, it dumps into a Word template. Instead of going into Crystal, we realized that, while we could cover most of our report wording guidelines, there would always be an exception, and we didn't want to have those exceptions be outside of LIMS.

Once the report is approved technically and administratively, it goes from what's called an assignments tab to a report tab. Once it is in the report tab, you cannot edit that file; you can only view it.

MS. BASHINSKI: That's what I was after, yes.

QUESTION: I just wanted to say that in Justice Tracks and in the Crystal report, you cannot make the changes once the report is finalized. Any time you make a change, it is recorded that you have changed it, so the changes are known. Plus, every analyst has his own password to put his report in and somebody else cannot edit it, only the administrator can do that.

MS. BASHINSKI: Any other comments, questions, thoughts, brilliant ideas?

QUESTION: I have a dream.

MS. BASHINSKI: A dream, oh, yes.

QUESTION: This is for Roger Kahn, talking about quality systems and making (inaudible). Everybody wants a quality system, but are you doing more than you should be doing? You said to meet the minimum standards but the minimum standards aren't necessarily defined. They're broad. So I'm wondering if something could be set up like a workshop where everybody says here are the minimum standards, here's what they do, how often it has to be done, what is the range, how often do you have to QC your balance, what is the temperature range, and how many days can you do it.

If everybody from the different labs sent in what they do and it was looked at, you could actually stop overdoing what you should be doing and at least meet the standard you have to meet for the system. I think that would be progress. I don't know where that would fit into NIJ or ASCLD, but what would be the minimum requirements for a quality system so that people wouldn't necessarily be overextending themselves doing quality control?

MS. BASHINSKI: Could that be something for the quality managers group to address? I mean, we have an organization.

DR. KAHN: Gee, I'm almost afraid to answer. Someone from a lab that I audited some time ago asked me yesterday about the NIST standard. The standard says you have to do the NIST SRM (standard reference material) annually; but the question is does it have to be done to every sample? It doesn't say.

Well, make it say it. So I promise you, if I write it or if somebody responsible for saying it is asked officially what you have to do, they'll say do them all. So do you want that?

You're not necessarily going to get the best answer by asking for a ruling. I'm nervous about that. I mean, it just seems as if when you have to put it down and say something that's beyond reproach that you end up leaving yourself a little margin for saying, I didn't try to choose the minimum.

I think if you're confident enough about standing by your work, then you can select your minimums. It really kind of varies with examiner and administrator confidence.

QUESTION: The age of the equipment.

DR. KAHN: The age of the equipment, yes. You have to know your system and yourself. The way it's supposed to work, I think, is that the extensions to the FBI Director's standards come via SWGDAM's quality assurance subcommittee and specifically through the audit document, because every little tweak gets there. As I understand it, there will be an update to the first one some time this summer or this fall. I know that it's almost done.

In a lot of cases, it just makes it more specific. You find yourself wondering who decided this and were things better off before the ruling. It was originally supposed to work through audit training sessions, where people would bring up their vague points for discussion, and some sort of recommendations would be funneled trough the SWGDAM's quality assurance subcommittee who would eventually put a comment into the audit document. So that's how you can influence the process.

MS. BASHINSKI: Okay. Well, I think we are done. I want to thank the panel very much for their help and their inspired comments.