Marine floating debris is a persistent issue in coastal waters, harming ecosystems, navigation, and shorelines. Effective monitoring is essential for rapid response and environmental management. However, debris in ports and bays appears intermittently and changes quickly due to tides, winds, and human activities. Traditional methods like ships, drones, or satellites often miss this short-term variability. Shore-based video cameras offer stable, frequent observations, making them a practical source for local monitoring. Yet automatic recognition from such images is difficult due to sun glint, waves, rain, wakes, and varied backgrounds. Moreover, debris types differ visually: plastic items are usually distinct with clear edges, while plant-derived debris forms irregular, diffuse patches. Using a single detection method for both limits performance.This study proposes a category-aware recognition framework using shore-based video images from Xiamen Bay, China. The dataset includes images under sunny, cloudy, rainy, and glint conditions, with both debris-positive and clean-background samples. The framework assigns different debris types to specialized visual branches. For plastic debris, an improved YOLOv8-based detector adds a P2 layer to capture fine details and uses an attention module to highlight relevant features. For plant-derived debris, a YOLO11-seg segmentation branch better represents irregular shapes that bounding boxes cannot handle well. The two branches are combined at the decision level using confidence scores, category-specific merging, and non-maximum suppression.Model evaluation included monthly tests, ablation studies, comparisons with other YOLO models, and false-alarm analysis on clean backgrounds. The framework achieved 88.85% average precision and 89.9% recall. Ablation results confirmed that the P2 layer and attention module improved plastic detection, while the segmentation branch gave more complete outlines for plant debris. Error analysis showed that sun glint caused most false alarms, while rain, low light, and complex wave patterns led to missed detections.These findings indicate that morphology-based task allocation effectively recognizes heterogeneous floating debris from shore-based video. The proposed system can provide candidate targets for manual review and routine inspections, supporting faster and more targeted coastal debris management. Future work will focus on temporal tracking and adaptive preprocessing to transition from single-frame detection to continuous event monitoring and operational regulation.