Fall Detection

The aim of this project is to detect dangerous events in live video streams, where people end up lying down on the ground for whatever reason (strokes, heart attacks, colliding with cars...). That should trigger an alert to make users check the image and call help if needed. It's terrible that even though they're right on camera, people die or suffer brain damage lying there low on oxygen just because no human is watching right then. To save lives, that kind of emergency recognition should be a freely available, easy to use and low overhead (ideally local) functionality for everybody who live-streams and monitors surveillance camera footage, and of course for those who write and distribute software to that end.

Computer vision and machine learning do have a heavy academic workload but when focusing on the actually working and up to date solutions in the offer you can still learn and implement pretty much in parallel and that is a lot of fun. That fall detector is a good opportunity to look at anything from more traditional computer vision to the latest neural network based visual recognition and just try it if it fits, without too much front-load. Unfortunately I stalled a few weeks ago mainly because I don't have enough images to train the neural network to achieve higher accuracies (with much less false positives) and can't think of an easy way to get them right now.

So here is the flow so far: the python code streams frames, at first to track motion. At that stage it would probably be a good idea to make sure we are looking at a moving human. But only if that's reliable and cheap enough for real time evaluation. The HOG for recognizing passersby that comes with OpenCV is fast but misses too much. Tapping a winning imagenet model to whitelist categories of persons could work, but I didn't have enough memory to implement (those models are huge) and am doubtful that it would deliver in real time.

Now assuming it's a person we are tracking, the fact they are moving usually implies they're walking on their feet and the rectangle surrounding their motion has a "tall" ratio (sides longer than base and top). Another assumption is that falling down should be a significant and sudden movement indicated by a somewhat large numerical difference between two frames.

So first we wait until two frames differ a lot, because this might be due to a person falling. The next test is for ratio changes: should they fall down and the movement tracking works perfectly and uninterrupted, their once tall ratio changes such that top and bottom become longer than the sides as they collapse and hit the ground. Or the motion tracking fails and the original rectangle decays into a weird cluster of scattered rectangles of different ratios while the person is going down. In that case chances are very high that at least one of those noisy rectangles has top and bottom longer than the sides, which is also good enough.

Partly relying on pure coincidence is not exactly an engineering feat but I suspect it will work just fine compared to adding any number of more sophisticated layers of hard coded tests you could think of - whatever you do adds some kind of error you'll be trying to cancel in the next layer and so on, and - come to think of it - that's precisely what machine learning is for.

But still you can't just have every frame checked by the neural network because even simple model estimates are too slow for that. And even if they weren't, many many false positives on frames you should not be evaluating in the first place would soon kill the vibe.

So only the image section where a suspicious ratio change following a big motion happened gets cropped and then evaluated by a neural network that is trained to recognize people lying on the ground. If our threshold is exceeded, we fire an alert. This can be done in real time.

You can see true positives in the video below - the thing is actually working and that is something I am genuinely happy about.

But I have a hunch that in the real world false positives would make most users switch the recognition off, seeing how those errors would propagate through a sequence of hundreds if not thousands of frames, so I think it is still useless practically.

The network just isn't accurate enough. It already tests over 80%, but that's on an insignificantly small number of images I reserved for testing. I only have a few hundred training images, pretty much everything google images finds for the keywords I could come up with. Aggressive augmentation did not cut it either. Maybe the diversity of positives is difficult to abstract upon. People can lie there in many different angles and directions, with or without obfuscation by things and passersby and helpers, face down or face up and what not.

By the way before building the neural network I did train haar-like and LPB classifiers with mostly the same images, resulting in false positives with a clearly prohibitive frequency. But I remember the haar-like, when true positive, seemed sensitive to how unconscious people on the ground spread their arms and legs - that group of features still seems promising to me.

I was thinking about buying a camera and feeding pictures of myself to the network but then those would not generalize too well to the rest of mankind. Manually taking pictures of others I might ask to lie down against different backgrounds would take quite a lot of real life time. Also I expect diminishing (accuracy) returns adding those new pictures to the model. And probably somewhere in the tens of thousands of images I would start needing proper GPU.

Anyway here is the python code that learns the model ('vanilla' convolutional neural network via keras, if you are working on 32bit windows you'll want the theano backend). I trained only briefly, maybe less than 20-30mins on a slow machine, trying to find the number of epochs right before overfitting starts, with different settings each competing for the highest test accuracy.