Malaria remains one of the most devastating infectious diseases around the globe, infecting 200 million people and resulting in over half a million deaths every year. Increasing resistance by malaria parasites to currently used antimalarials across the developing world requires timely detection and classification so that appropriate drug combinations can be administered before clinical complications arise. Here, we developed a simple, inexpensive and portable image-based cytometer that detects and numerically counts the malaria infected red blood cells (iRBCs) from Giemsa-stained smears. The developed cytometer is able to classify parasitic subpopulations based on the quantification of the area occupied by the parasites within iRBCs and demonstrates high specificity and sensitivity in calculating the parasitemia irrespective of its developmental stage. Moreover, selected antimalarials were tested using our image-based cytometer in comparison to commercial flow cytometer, which demonstrated comparable and matching results with the previously published results. Collectively, these results highlight the possibility to use our portable image-based cytometer as a field-deployable tool for cheap, rapid and accurate malaria diagnosis and antimalarial testing without compromising on the efficiency.