BE TESTY or: are your software results incomplete?
No matter what investigative, forensic or analysis discipline you are in, please be sure to test and be comfortable that your software produces true, accurate, repeatable and complete results for your needs.
How do you know the software you are using is producing true and accurate results?
A great question for attorneys to ask you. What is your answer.
If your answer is, it was listed on the web as useful "insert your topic here"
software, "others use it", and have recommended its use. Well, we know if its listed on the web, it must be useful. If you use any of these arguments, I think your case is going south.
Probably the best answer would include something like, I have tested it myself to determine if it satisfies my needs for this particular investigation. The key words here, are "tested it myself", and "for this particular investigation".
Suppose you are performing string searches. You may obtain a string search program from a "reliable" source. (note at this time, I believe NIST is performing or setting up test environments for string search programs. Verify this statement for yourself. If you take my word for it, I have a bridge in Brooklyn that is for sale). You obtain and run the string search program and it shows no results. So your report says, "didn't find any suspect strings". What you didn't think about was, are your strings stored in an unusual format (unicode UTF-16, in a compressed format, unusual syntax, etc), and is the program designed to find those unusual items.
You may have known that the strings were stored in your data in an usual format, but what you didn't know (or test), is that the string search program wasn't coded to search for those formats. You just "ass"umed the program would work for your needs. Not good for evidence and never lie in court!
Another example, (and I admit, I'm not a network investigator, in fact, I can barely spell ntwrk). Anyway, suppose your software was built to search on only IPV4 addresses, but it wasn't documented that way. And your analysis needed to analyze IPV6 formats.
You may not find any hits, but you didn't know the software wasn't coded to find the particular network problem you are searching for.
Bottom line: Test your software. Test it to determine if it produces results for the particular subject of the investigation. Test it at both ends of the bell curve. Don't be a ding-a-ling and fail to ring the bell. Don't rely on the software writers claims
or other users that it works in all instances.
If you do, you may be missing important data and leave yourself open to challenges and possibly loss of evidence value.
(hopefully), You know what you are looking for .
(hopefully), You know how to create some test data.
Why don't you create your own test data for the specific items you are searching for.
In my previous life, when we found no hits, we knew something was wrong, so I learned to "SEED" a test platform. This will allow you see if your software can find it.
If it will, then it will probably find the data on the suspect platform.
Very few programs will produce 100% accurate results for all instances.
Does your choice of software, produce accurate results for what you are looking for.
Only your tests can confirm this.
And will help you answer the attorneys question, Did you test the software?
You may have and probably will have to to testify to that fact. Even though my software doesn't contain bugs, in some instances its just operationally challenged. As is the situation with many software programs. But, is the place(s) where the software may fail, or have its restrictions, a place that would raise questions to your specific investigative results.
I don't really care that my software will fail at the petabyte level. Because I know it will probably never be used on petabyte files. But it does work on gigabyte files.
Can you say that. And have you tested!
Go to www.dmares.com for amny programs that HAVE BEEN TESTED!
Author - Dan Mares. Dan is a respected Friend and very knowledgeable digital forensic investigator.
Dan Mares founded Mares and Company, LLC in 1998 after retiring from a 27-year career as a federal law enforcement agent. During that time he became interested in and obtained training in computer science. He began developing software programs designed to analyze large amounts of data retrieved from mainframes. Those programs were the precursors to the current Maresware data analysis software. Around 1986, he began working in the area of what is now termed 'computer forensics.' In the search for tools more suitable to the specific needs of computer forensic investigations, he began developing software that was later called Maresware computer forensic software.While serving as a federal agent, Dan assisted in the development of the Seized Computer Evidence Recovery Specialist (SCERS) course at the Federal Law Enforcement Training Center in Glynco, Georgia. He also served as a guest SCERS instructor. He also assisted in the development and teaching of the Basic Data Recovery and Advanced Data Recovery classes at the National White Collar Crime Center.A few of the organizations he has appeared before as guest speaker include: International Association of Computer Investigative Specialists (IACIS); University of Texas, Austin.; Kennesaw State College; U.S. Secret Service; FBI Academy in Quantico, Va.; High Technology Crime Investigation Association (HTCIA); and Norwegian National Police Academy.