Man, it's going to be great when this gets adapted to make sure I'm looking into the screen at all the ads I'm required to watch, or when it complies a report of whether or not I'm paying attention to my boss in an all-hands...
Others are already trying to do this https://arxiv.org/abs/2504.06237. I haven't seen anyone take the approach I tried though, as most uses cases focus on tracking the main user rather than others around.
> It’s an application which detects people looking at your screen. The aim is to keep you safe from shoulder surfing, utilising your webcam to give you the power to prevent snoopers.
When is this ever a problem that cannot be solved by positioning yourself with a wall behind you or going somewhere private? This feels like overkill for the stated use-case. I can imagine someone thinking they might need this to do private stuff in a public space (a coffee shop?), but they'd turn paranoid from everyone passing by just glancing around.
Also, is this a realistic threat model anywhere? People snooping by standing behind you tend to be colleagues or totally random passers-by; not people actually interested in gleaning private information. Anything more serious than logging into your Facebook account would imply simply having proper OpSec procedures (like: 'only do this in private').
All I can think of is employee monitoring where such tools will just end up making people insecure in their workplace (and less productive, because gazing out of a window or into nothingness actually helps when you are doing work which requires pondering; and less healthy, because looking away from your screen into the distance is recommended for anyone with working eyes).
So much text and not a single example, diagram, or demo.
I'm honestly skeptical this will work at all, the FOV of most webcams is so small that it can barely capture the shoulder of someone sitting beside me, let alone their eyes.
Then what you're basically looking for is callibration from the eye position / angle to the screen rectangle. You want to shoot a ray from each eye and see if they intersect with the laptop's screen.
This is challenging because most webcams are pretty low resolution, so each eyeball will probably be like ~20px. From these 20px, you need to estimate the eyeball->screen ray. And of course this varies with the screen size.
TLDR: Decent idea, but should've done some napkin math and or quick bounds checking first. Maybe a $5 privacy protector is better.
Here's an idea:
Maybe start by seeing if you can train a primary user gaze tracker first, how well you can get it with modeling and then calibration. Then once you've solved that problem, you can use that as your upper bound of expected performance, and transform the problem to detecting the gaze of people nearby instead of the primary user.
Sorry, I haven't gotten around to gathering examples yet. I ran the models on some example videos which is where the accuracy stats come.
Perhaps I have been jaded by the Mac webcam, I agree on most old webcams it wont be great but on newer webcams I have had success.
I did try a calibration approach but it's simply too fragile for in the wild deployment, calibration works great if you only care about one user but when you start looking at other people it doesn't work so well.
Good idea, it may be more fruitful to do that. At least then for the primary user we can be much more certain.
This models supports the EyesOff application which will prevent information extraction by either having a popup, switching to another app, or a notification(you can define the behaviour in a few different ways).
Privacy screens are still useful and I recommend people to use EyesOff and the screen protector. A privacy screen won't stop someone shoulder surfing from directly behind you etc.
Yeah tbh I do recommend using this alongside a privacy screen for best protection. Privacy screens also suffer from the fact that they won’t block someone directly behind you from seeing the screen, so both methods have issues.
Any tips on improving accuracy? A lot of it might be due to lack of diverse images + labelling errors as I did it all manually.
https://github.com/rehg-lab/eye-contact-cnn
When is this ever a problem that cannot be solved by positioning yourself with a wall behind you or going somewhere private? This feels like overkill for the stated use-case. I can imagine someone thinking they might need this to do private stuff in a public space (a coffee shop?), but they'd turn paranoid from everyone passing by just glancing around.
Also, is this a realistic threat model anywhere? People snooping by standing behind you tend to be colleagues or totally random passers-by; not people actually interested in gleaning private information. Anything more serious than logging into your Facebook account would imply simply having proper OpSec procedures (like: 'only do this in private').
All I can think of is employee monitoring where such tools will just end up making people insecure in their workplace (and less productive, because gazing out of a window or into nothingness actually helps when you are doing work which requires pondering; and less healthy, because looking away from your screen into the distance is recommended for anyone with working eyes).
I'm honestly skeptical this will work at all, the FOV of most webcams is so small that it can barely capture the shoulder of someone sitting beside me, let alone their eyes.
Then what you're basically looking for is callibration from the eye position / angle to the screen rectangle. You want to shoot a ray from each eye and see if they intersect with the laptop's screen.
This is challenging because most webcams are pretty low resolution, so each eyeball will probably be like ~20px. From these 20px, you need to estimate the eyeball->screen ray. And of course this varies with the screen size.
TLDR: Decent idea, but should've done some napkin math and or quick bounds checking first. Maybe a $5 privacy protector is better.
Here's an idea:
Maybe start by seeing if you can train a primary user gaze tracker first, how well you can get it with modeling and then calibration. Then once you've solved that problem, you can use that as your upper bound of expected performance, and transform the problem to detecting the gaze of people nearby instead of the primary user.
Perhaps I have been jaded by the Mac webcam, I agree on most old webcams it wont be great but on newer webcams I have had success.
I did try a calibration approach but it's simply too fragile for in the wild deployment, calibration works great if you only care about one user but when you start looking at other people it doesn't work so well.
Good idea, it may be more fruitful to do that. At least then for the primary user we can be much more certain.
Privacy protector solves different problems - they prevent people from extracting information on screen, not merely inform about possible infraction.
That being said it's useful in a way that if I'd see anything like that in a contract it wouldn't be a red flag. It'd be red flashing GT*O alarm ;)
Privacy screens are still useful and I recommend people to use EyesOff and the screen protector. A privacy screen won't stop someone shoulder surfing from directly behind you etc.
There is also better ways to do this sort of task when all you care about is tracking the main user: https://arxiv.org/abs/2504.06237, https://pmc.ncbi.nlm.nih.gov/articles/PMC11019238/
Interesting problem anyway. I'm surprised the accuracy is so low.
Any tips on improving accuracy? A lot of it might be due to lack of diverse images + labelling errors as I did it all manually.