Jakub Langr's Blog

Posted in technical

python AI reviews StartUps

Lots of people are arguing about the theoretical nature of what AI means many years down the road, but I would like to focus on the AI commercial products in the next 2-5 years. I'd also bring all the incredible, but deeply technical advances back to the real world and business and focus on and focus on what businesses and consumers can practically expect of the future. First off I will talk about the most interesting advances in AI, second what the current applications of those advances are and wrap it up with what this means for business.

At the same time, this is not intended to be a discussion of the business theory behind AI-startups. This has been covered by more qualified people, for example here.

1. Breakthroughs in AI

Last couple of months were excellent for AI: everything from One-Shot-Learning, advances in Generative Adversarial Networks (GANs) and Memory-Augmented Networks to practical applications such as AlphaGo, Liberatus winning the world Poker championship and a neural net creating images that match textual input.

There have been way too many practical applications to list, but all of these are a results of more theoretical work from prior years. Therefore, in order to look at what future is likely to hold, we should look at the theoretical advancements and extrapolate what kind of practical tasks these novel techniques can achieve in production. One such area is reasoning AI: memory augmented and differentiable networks are likely to bring in a new era of higher level reasoning: as recent Stanford papers and recent Facebook tools to help visually impaired understand content of images demonstrate. So reasoning and Q&A AI is just around the corner. In terms of future theoretical breakthroughs, the most fruitful areas are likely to be:

Further improvements on GANs, as they already are being tested in the field of 3D object generation, remastering old movies and image enhancement. Even one of the fathers of deep learning, Yann LeCunn, called it “the most important [recent advancement in deep learning]”.

Further improvements on memory/attention as demonstrated for instance by a recent study by DeepMind published in Nature.

2. Summary of state of the art, future improvements and likely future paths

The biggest breakthrough, in line with LeCunn

“This approach [GANs for semi-supervised learning] allows us to obtain state of the art results on MNIST, SVHN, and CIFAR-10 in settings with very few labeled examples. On MNIST, for example, we achieve 99.14% accuracy with only 10 labeled examples per class with a fully connected neural network — a result that’s very close to the best known results with fully supervised approaches using all 60,000 labeled examples. This is very promising because labeled examples can be quite expensive to obtain in practice.” link

To summarize: this means that people can get performance as good as the best networks in the field trained with the full dataset.

I foresee that the next improvements will apply similar techniques to more complex datasets and using fewer and fewer labelled examples to match the state-of-the art performance. This can probably be realized by not only using transfer learning as the “training agent” more effectively (or using multi-task learning), but also by using advanced GANs themselves, as hinted to above. Or maybe even to move closer to optimal Active Learning by generating exactly the hardest (real) examples for the network to classify at that point in time to improve the fastest. (This does not depend on advances in other areas, such as Synthetic Gradients, better optimizers etc.)

In other words, now we are advancing, in a sense, in the "theory of knowledge" of Machine Learning algorithms so that they can learn faster. Akin to a child in middle-school, deep learning is now discovering "learning aids" or "hacks" to learn even faster and generalize better. This means that we can move AI closer to production and usefulness. This means that plugging into a new (business) vertical will be easier than ever. You will still need to develop vertical-specific expertise and architecture, but technology like generic object detection, speech synthesis or sentiment analysis will be commoditized.

3. Commodtizied ML is here, so what?

These advancements along with better tools for AI researchers and production, will favor companies who have (i) unique business partnerships or (ii) overcome some unique regulatory hurdle. Huge opportunities are still in FinTech and other regulation-heavy industries.

However, the key point is to realize that the AI of the future will likely just serve as a preprocessing or feature engineering step feeding into the next, traditional data science, layer of algorithms. But make no mistake: this is huge. Computers, for the first time in history, will have the ability to see and create their own representations of objects. To take a practical example: I have recently been through an accelerator where one of the companies, Observe, is working on automating fish farming. They have a standard object recognition algorithm that passes on its output to further systems managing feeding of fish etc., so that they maximize the fish stock and minimize waste. Their key selling proposition in this case is the second, fish feeding, algorithm, but until couple of years ago there was no easy way to explain the images of fish to the second (control) algorithm.

To take a traditional computer science analogy, we can think of the case of John Searle's Chinese room experiment: so far the computer was just manipulating symbols with no conception of what those symbols mean. I.e. the computer could not be thought of as "understanding" something or being "conscious". I do not want to delve into the philosophical implications of this statement, but the core dilemma is: because the computer has no context, it cannot be thought of as operating on anything else than a (computationally) straightforward set of instructions (no matter how complex in human terms). Now, computers are getting more context, so, in an odd sense, we are closer to understanding.

From Wikipedia under CC license.

Keeping in line just with the practical sense, computers will now be able to auto-extract features. In fact, for instance Eyn's core tech is already a facial detection algorithm fed into spoofing detection. This is only possible because facial detection is so close to perfect we can rely on it almost as deterministic. Andrew Ng talked about the importance of moving from 95% to 99%, I think there is another, perhaps even more important milestone in moving from 99 to 99.9% where we can use commoditized machine learning almost as a framework for our core tech.

This is especially important in parts of the world where databases about the real world are not as common as in the EU. Because algorithms can now integrate into physical space and get data from it directly, we can build directly onto the physical space: i.e. we can provide value to all of the ~6.1 billion people currently underserved by technology and apply technology directly to areas they find most useful, like agriculture: Optimal Labs is already doing that. Moreover, with all this computer vision, it will be much easier to jump-start projects like city planning: in fact, this is already happening in India.

The last thing I would like to talk about is a direct implication of the previous paragraph, but has far reaching consequences. For the first time, IT systems may make sense even in situations in which there is no other IT system to connect to or IT support. Algorithms now do not need supporting database--as they can create their own or rely on pre-trained models--so they can be rolled out to a completely new company without the usual enterprise sales & integration cycle. In theory, you can have an entire IT system managed by external companies as it is quite expensive to have AI talent or ML maintenance on site (e.g. in agriculture) beyond simple mechanics. Now, this is wildly speculative, but if true, this absence of in-house IT people means that external companies can deploy to completely novel areas (and there'll be more of those opportunities because in-house IT is not there to pitch obvious technology solutions) and capture much more of the market. But it also means that there are no longer IT people to guide and support you so the business contacts are now more important than ever.

Conclusion

So from the recent advancements and meta-learning theory it is clear that we are in an age of vertical AI and that the future direction is likely to heavily rely on understanding the business use-cases of particular AI technology. This seems to be further accentuated by the fact that AI can now be deployed on its own without external support or databases to connect to. This means that the business side of AI will be more important than ever, but also the opportunities are becoming more plentiful, because even the very obvious usecases might not be covered by in-house teams.