COMPARISONS of DM Tool Discovery
with other well known DM methods
Financial forecasting
The results of "Discovery" Tool comparisons with Neural
Networks (NN), Decision trees (Sipina), Rules extracted from NN,
Firstorder logic methods (FOIL) and other benchmark methods presented
on the table and figure
Breast cancer diagnostic system
The results of comparisons with Neural Networks, Decision
Tree (SIPINA), Linear Discriminant Analysis, "SIGAMD" software
Figure presents results
for another selection criterion: level of conditional probability. We studied
three levels: 0.7, 0.85 and 0.95. A higher level of conditional probability
decreases the number of rules and diagnosed patients, but increases accuracy of
diagnosis. Results for them are marked as MMDR1, MMDR2 and MMDR3. We extracted
44 statistically significant diagnostic rules for 0.05 level of F –criterion
with a conditional probability no less than 0.75 (MMDR1). There were 30 rules
with a conditional probability no less than 0.85 (MMDR2) and 18 rules with a
conditional probability no less than 0.95 (MMDR3). The total accuracy of
diagnosis is 82%. The false negative rate was 6.5% (9 malignant cases were
diagnosed as benign) and the false positive rate was 11.9% (16 benign cases
were diagnosed as malignant). The most reliable 30 rules delivered a total
accuracy of 90%, and the 18 most reliable rules performed with 96.6% accuracy
with only 3 false positive cases (3.4%). Neural Network (“Brainmaker”, California
Scientific Software) software had given 100% accuracy on training data, but for
the RoundRobin test, the total accuracy fell to 66%. The main reason for this
low accuracy is that Neural Networks (NN) do not evaluate the statistical
significance of the perfect performance (100%) on training data. Poor results
(76% on training data test) were also obtained with Linear Discriminant
Analysis (“SIGAMD” software, StatDialogue software, Moscow). The Decision Tree
approach (“SIPINA” software, Universite Lumiere, Lyon, France) performed with
accuracy of 76%82% on training data. This is worse than what we obtained for
the MMDR method with the much more difficult RoundRobin test (fig. 8). The
very important falsenegative rate was 38 cases (MMDR), 89 cases (Decision
Tree), 19 cases (Linear Discriminant Analysis) and 26 cases (NN).
In these experiments, rulebased methods (MMDR and decision trees) outperformed other methods.
Note also that only MMDR and decision trees produce diagnostic rules. These rules make
a computeraided diagnostic decision process visible, transparent to
radiologists. With these methods radiologists can control and evaluate the
decision making process. Linear discriminant analysis gives an equation, which
separates benign and malignant classes. For example, 0.0670x10.9653x2+…
represents a case. How would one interpret a weighted number of
calcifications/cm 2 (0.0670x1) plus a weighted volume (cm 3), i.e., 0.9653x2?
There is no direct medical sense in this arithmetic.
