Automated Marine Species Classification and Segmentation at Fish Markets Using Convolutional Neural Networks and Transformer Architectures

Introduction

An automated method for the classification and segmentation of marine species is presented in the context of the Burriana (Spain) fish market. Using convolutional neural networks and pre-trained transformer-based models, this approach aims to optimize species identification and size estimation, improving the efficiency and objectivity of fishery resource management. The ultimate goal is to digitize marketing workflows in fish markets, enabling real-time online auctions as fish arrive at the market.

Methods

A dataset of fish tray images was acquired at the Burriana fish market between February and July 2023, organized according to FAO species codes, and manually curated to exclude mixed-species samples. Balanced datasets of the most commercially frequent species were generated using hold-out training, validation, and test splits. Species classification was performed via transfer learning with pre-trained deep learning architectures implemented in TensorFlow/Keras: EfficientNetB7, ResNet152V2, MobileNetV2, NASNetLarge, and InceptionResNetV2. Data augmentation was applied to enhance generalization. For segmentation and size grading, the Segment Anything Model (SAM) was used to generate fish masks, followed by post-processing filters based on area, overlap, and contour similarity. Performance was evaluated using accuracy, precision, recall, F1-score, Average Precision (AP), and Intersection over Union (IoU).

Results

InceptionResNetV2 achieved the highest classification performance, reaching 99% test accuracy across 16 marine species. SAM attained a mAP of 89.9% and an AP score of 79.6% at IoU = 0.90 for monkfish segmentation, enabling objective size grading. Mixed-species experiments confirmed the model's capacity to identify dominant species under complex market conditions.

Conclusions

The proposed methodology demonstrates the potential of deep learning and transformer-based computer vision for automated species identification and size grading in fish markets. These results support future applications in the digitalization, traceability, and automation of fishery commercialization processes.