Multispectral remote sensing images classification is a challenging task due to the complexity and variety of natural scenes. Deep Learning has revolutionized the field of remote sensing by achieving impressive performance. However, its effectiveness relies on the availability of large labeled datasets, which can be hard to obtain. Recently, Transfer Learning has been proposed to resolve this issue by leveraging pre-existing knowledge from pre trained models on millions of raw data. Some attempts of transferring knowledge from computer vision to remote sensing have achieved acceptable results. However, those models may miss the rich spectrum information necessary for precise results and are not well suited for multispectral imageries. Additionally, transfer learning in remote sensing applications is extremely dependent on the selection of data from the source model, which has a significant impact on the target model. This study investigates the importance of selecting the appropriate source dataset and task for transfer learning. To this end, pretrained model on Landsat imageries have been finetuned on sentinel-2 data. Separated pre-trained models on sentinel-2 imageries for palm/building classifications have been employed to explore the impact of the source classes on the performance of the target class. The results have been evaluated by metrics such as Precision, Kappa, Recall and F1-score to assess the model's performances where results from train/finetune on similar data source (sentinel-2) reaches higher accuracy compared to multisource training/finetuning. This study illustrates how selecting appropriate source datasets and tasks can directly influence the performance of the target model.