I work in machine learning for healthcare and people often come to me suggesting an expert system. Usually what they mean is a hand-crafted version of a decision tree. I haven’t been able to find any good articles really articulating the cost/benefit of each approach with examples and I’m wondering: What has everyone else’s experience has been?

Examples I usually give:

• One part of the core algorithm in an app was a giant decision tree from ~5-10 years ago, 3-4 deep with very complex conditions. One of the smartest guys around decided to understand and document the code. Even after several days of experimentation and talking to the author, he couldn’t explain it all. Due to the feeling that it was important, we weren’t able to remove the unknown code and therefore could clean it up either. That code was doing pattern matching for mobile keyboards – would’ve been a good fit for ML.
• Another project would jump-start text query parsers with a hand-written grammar. Once it seemed reasonable they’d release it, collect query data, annotate it, then replace it with an ML system. That meant a much faster time for both the first and second releases.
• I’ve seen a great example in a book but I forget which – it talks about classifying something as a bird or not and demonstrates the challenges of building a high quality system.