Sentinelle

Context

Sentinelle explores an applied research question: can we estimate a healthcare professional’s cognitive load continuously, without an intrusive sensor — and offer the right help at the right moment?

Problem

Cognitive-load measures usually rely on dedicated sensors that don’t fit everyday clinical use. The system needed an estimate from lightweight signals (voice, face), and then an assistant able to change strategy based on the detected state: you don’t advise a relaxed clinician the same way as an overloaded one.

Architecture

• Audio and video signals: sliding-window acoustic features with librosa (MFCCs, fundamental frequency, energy, jitter, speaking rate) and visual features (posture, head pose) with MediaPipe.
• Cognitive-load models:
  • a CNN-LSTM fusion model — a MobileNetV2 visual encoder and an audio branch joined in a shared LSTM — trained on RAVDESS (audio + visual);
  • a task-level MLP on AVCAffe, a cognitive-load dataset, with per-participant normalisation, benchmarked against Random Forest, SVM and XGBoost;
  • CLSE, PoseTrack and BIWI as supporting datasets, to validate audio-only, posture and head-pose features.
• Triggering: the agent only activates when the load score crosses a critical threshold; otherwise it just monitors on the dashboard.
• Clinical RAG agent: WHO guidelines split into chunks, embedded with nomic-embed-text and stored in ChromaDB; a LangChain RetrievalQA chain with Mistral 7B (Ollama, quantised).
• Load-aware prompts: three prompt templates — high load: immediate interventions, patient safety first; medium: monitoring and prevention; low: learning and complex tasks.
• Voice interaction: the clinician’s speech is transcribed with Whisper, and the sourced answer is shown on the dashboard and read aloud (Piper).
• Fully local: no external API; the index persists and the knowledge base can be updated without retraining the model.
• Infrastructure: Docker Compose — FastAPI, Ollama, ChromaDB, MLflow for experiment tracking, Redis + Celery for long-running tasks, PostgreSQL for metadata, MongoDB for raw artefacts — plus GitHub Actions CI.

Stack

Python, PyTorch (MobileNetV2, LSTM, MLP), scikit-learn, XGBoost, librosa, MediaPipe, Whisper, LangChain, ChromaDB, nomic-embed-text, Mistral 7B via Ollama, Piper TTS, FastAPI, Celery, Redis, PostgreSQL, MongoDB, MLflow, Docker, React + TypeScript. CNN-LSTM trained on Kaggle GPUs.

Highlights

• End-to-end chain: from the clinician’s voice and image to a sourced clinical recommendation, read aloud.
• An agent that adapts: one knowledge base, three response strategies depending on the detected cognitive state.
• Research approach: several model families benchmarked, participant-grouped cross-validation so the model can’t just recognise the person, and RAGAS evaluation of the RAG agent.
• Privacy by design: everything runs locally; no data leaves the machine.
• FastAPI backend and React dashboard in progress.