detecting diseases in high resolution endoscopy videos can be done in several ways depending on the methodology for detection. one such method that has been a hot topic in the field of medical technology research is the implementation of machine learning techniques to aid in the diagnosis of networks. while, this has been studied extensively with traditional machine learning methods and more recently neural networks, major issues persist in their implementation in everyday health. among the largest issues is the size of the training data needed to make accurate prediction, as well as the inability to generalize the networks to several disease. we address these issues with a novel approach to detecting inflammatory bowel diseases, specifically crohn’s disease in endoscopy videos. we use thick data analytics to teach a network to detect heuristics of a disease, not to simply make classifications from images. using heuristic annotations like bounding boxes and segmentation masks, we train a siamese neural network to detect video frames for ulcers, polyps, erosions, and erythema with accuracies as high as 87.5% for polyps and 77.5% for ulcers. we then implement this network in a protype frontend that physicians can use to upload videos and receive the processed images in an interactive format. we also pontificate as to how our approach and prototype can be expanded to several diseases with learning of more heuristics.