A new test of plagiarism detection systems
The research group at the FHTW has again tested plagiarism detection systems. We have extended our collection of test cases and now also have a selection of collusions - slightly changed papers that strongly resemble other papers - so that we can check how well software can find collusions. We also included a paper stored in a closed database (Springerlink) and one which we copied by typing up a page we found on Google books. An additional nasty test case replaced the letter "e" in a paragraph with the letter "ε". It still looks okay at first glance, but many systems were not able to find the plagiarism.
The results are, as always, not exciting. Software just cannot find translation plagiarisms or plagiarisms that are taken from books. Teacher, however, are often very good at spotting "fishy" texts like this and can even find sources using search machines. Even removing all the test cases with this kind of plagiarism from our evaluation, we still had no software that was given the grade of "very good". The systems have gotten slightly better, however.
We also evaluated the usability of the systems - many have lots of problems in this area. A particular problem is the numbers reported - it is often not clear, what the numbers or percent values given mean. And the reports are sometimes not very useful or the system is difficult to use in a university setting. Having to submit papers one at a time and having to wait a longish period between each test is not acceptable. |
