How do overcome these obstacles and what do you do to avoid encountering the same problems in the future?
The hardest part about early stage innovation is being emotionally capable of killing your ideas so that you can spend your time pursuing other ideas that might work. You cannot fall in love with your ideas during early stage innovation.
The best way to avoid this problem is by having lots of ideas that you can pursue if the current one stops looking appealing. I know that's not easy, but it works.
Companies often suck at this. They want to be innovative and so they send off a team of smart people to work on a really hard problem that has a significant risk of failure. So far, so good. But if this team doesn't feel like failure is an option, then things get ugly quickly. Now the team won't be honest about the challenges they're having. They won't tell management, "Hey, we gave this our best shot but we don't think the project is feasible." So instead they'll burn through money and time, struggling to solve something that can't be solved.
But, imagine if management said the following to the team...---Here are 10 potential projects that we think are really hard. We want you to review them all and allocate time at your discretion to see which are worth pursuing. We don't expect you to succeed in making any of these ideas work, but if you do, we think it will be a huge win for our company and a benefit to the world. So we're willing to take this risk. You have xx months and a rough budget of $___.
As you work on the ideas, you're going to discover that some of them are impossible, won't have the impact we expected, or are otherwise not worth pursuing. We look forward to reviewing your analysis that leads to those conclusions so that we can kill those ideas. That will also be considered a win because by killing the ideas effectively, you'll save our company tons of money, time, and pain working on something that would have been destined to fail.----
Wouldn't this be a better approach? Wouldn't you want to be part of THAT team?