Sensor-Rich Smart Personal Photo Organization

With the popularity of visual sensor on mobile devices, that is the cameras, it becomes a habit for many people to take photos everyday and everywhere. This led to the rapid expansion in number of personal photos and becomes a problem to the users in storing and organizing them, which had not been faced before. Fortunately, cloud storage provided a comprehensive solution at the right moment, and it facilitates the synchronization and sharing of photos acquired. However, organizing this large number of personal photos is still a tedious and painful task. Common needs in photo organization may include tagging, removing duplicated or similar photos, and grouping photos into albums. In this paper, we target to provide a smart personal photo organizing application which make use of existing sensors and related technologies to help users to handle duplicated or similar photos more efficiently. For example, photos can be sorted, reorganized or searched according to different information, like geographical, time or appearance, acquired from various sensors on the mobile device. By harnessing the power of cloud computing for image analysis and face recognition algorithm, our system significantly reduce the time spent on managing photos in a neat and tidy way which reduce user frustration and enhance user experience.


Introduction
Digital camera is already a standard component on every mobile devices nowadays, including smart phone, tablet and notebook.Many people are starting to rely on their smart devices for daily photo-taking instead of using a standalone digital camera.There is a trend that people are taking more and more photo each day because of this convenience.As a result, the number of photos taken or obtained from friend's sharing can soon create a great problem on where to store and how to organize them from various sources.For example, it is difficult to find old trip photos in one's photo collection, as the photo album is overwhelmed by many newer photos added daily.Also, duplicated images do exist because multiple friends are sharing the same photo with various quality or resolutions.Except that the user spend extra time in organizing these photos well before, but it is usually a tedious task.
Although most users nowadays are still sticking to traditional media for storing their photos, like mobile device storage, harddisk, and memory cards; many of them have started moving to cloud storage for backup or easy of access of this large amount of photos from different platforms.Existing mobile devices are capable to work on the processing of images, however, it is rather limited to single and medium sized photo.The computational power of mobile devices is still difficult to help organizing a large number of photos quickly and efficiently.
Therefore, in this project, we are going to develop a photo organization application that facilitates the management of photo on mobile platform.We attempt to analyze information collected from various sensors accompanied with the mobile device, for example the geographical location, time, weather and appearance.Based on the similarity of these properties, our system will provide useful functions to tackle common problems encountered.For example, we will identify duplicate photos that may be gathered from different sources.This commonly occurs when we share share photos on the Internet or other family members, and multiple copies appears in different repositories.Thus, these duplicated photos can have different resolution, compression quality and etc, while their contents are the same.Our system also provides sorting and grouping capabilities to the user, so that users can reorganize the photos in different ways based on needs.to facilitate the finding of desired photos when necessary.This is enabled by employing proper image features or auxiliary properties from the sensors.Therefore, we can reorder photos according to their similarity or search photos with resembling appearance.This greatly helps users to find desired photos easily and without the need to first organize the photos regularly.
In order to prevent relying on the mobile device for analysis, we shift many of the computational heavy tasks to the cloud.These include the computation of image similarity feature, the face detection and recognition.As a result, the mobile platform works as a lightweight terminal to commit the instruction and display results.The project will benefit all users who want to better manage their photos.The smart personal photo organizing application can bring the convenience to the users, also aim to provide a simple and easy way for users to view and manage their photos.
We have implemented a prototype system on the Android platform, and supported with a back-end Apache web server.We had carried out a user study to learn if the system is useful.Preliminary experimental result shows that the users are satisfactory to the photo organization platform and think that it can help to manage their daily photos easier.

Related Works
Content-based image retrieval had been a topic of research in academic and industry for decades [1,2].Usually, a critical issue for quality query result relies on the extraction of proper features and the related power in evaluating the similarity.Popular image similarity measures include color histogram [3], GIST [4], HOG [5], BoW with SIFT [6,7].
Later, the application moves to the seeking of Internet or Cloud images, which requires fast query and intention prediction [8][9][10][11].Thus, proper indexing of the images is necessary.When the mobile devices become more popular and widely available, the problem of organizing personal photos become apparent as user get used to take photos daily and the accumulated number of photos can increase dramatically within months.Existing mobile applications that help to organize user photos usually provide limited functions, like sorting by date and time.Furthermore, the photos sharing among friends is common, popular ways includes social networks, instant messengers and online albums.These shared photos can easily create redundancy in the personal photo collection.Some specific mobile apps can handle seeking of duplicate images, however, they are likely to find duplicate images that are encoding exactly, but not the same scene or contents.This may missed duplicated images that are having different resolution or compression qualities.To the best of our knowledge, not much existing works are dealing with the organization of personal collections via a mobile environment.While there are works about designing a novel user workspace for organizing and visualizing photo collections [12,13].Therefore, this paper is trying to fill this gap, and preparing to seek a solution of organizing some emerging problems of personal photo collections that are acquired from mobile devices or other sources.

Overview and Implementation
Our system is built on the cloud with the mobile app as client side, and a web application server which stored photo and auxiliary information.Figure 1 shows an illustration of the components in our mobile photo organizing system.The client mobile app provides a user friendly interface for searching and organizing similar photos in their collection.As mentioned, most of the related computations are delegated to the server side of the system.As all the photo are assumed to be taken from the mobile phone, the mobile application will automatically upload photos via the Image uploader module.
Therefore, both the client and server sides are having a synchronized collection of photographs.While the client copy is for display purpose, and the server copy is for analysis and reorganizing purposes.
There are a number of major components on server side, including the sorting module, searching module, geographical analysis module, image similarity analysis module, face detection and recognition module.As the similarity and face feature extraction involve computations that are relatively intensive when applied to a large number of images, it is impossible to perform them on-the-fly.Precomputation is required whenever a new photo is being synchronized on the cloud.On the client side, user can perform different organization functions including duplicate identification, similarity based sorting and searching.The corresponding module then communicates via web services with the server to retrieve a list of reorganized results.These web services communications are relatively light-weighted, as no photo content is being sent, but only the corresponding IDs of the photos.
We have implemented our prototype system on Android platform with Apache application server for the server side support.All photos are stored with JPEG compression.In the following, we will introduce some of the important components in our system individually in more detail.

User Interface and Operations
Our application assumes to allow user to manage their photos solely via the mobile interface.However, one significant problem of working on mobile device is the limited screen size.Similar to many existing personal photo browsing applications, our mobile app can display large number of thumbnail images in a grid based manner (Figure 2(a)) which tightly pack images as many as possible inside the limited screen space.When user click a particular thumbnail, the image will be enlarged to present a fine detail view to user as in Figure 2(b).
User can easily choose sorting the images in a number of ways from the top menu bar.From the drop-down menu in Figure 2(c), users can select to sort by date, location and face.The corresponding sorting result can be found in the result section, or directly from Figure 4. Searching for similar images from the personal collection is also convenient from the interface.After the user selected an image, then a contextual menu will appear as in Figure 2(d).Apart from search based on similarities in appearance, our system also support search by face.

Geographical Analysis
Photos captured on mobile devices are likely to contain geographical tagging information.Usually, they recorded the longitude and latitude of where the photo is taken.To enable a useful sorting by location of photo taken, we applied clustering on the whole collection.With the use of meanshift algorithm, photos acquired at the same place or near locations will be grouped in a cluster.As location has no natural origin point, the resulted sorting by location will display these clusters one after another in an arbitrary manner.

Image Similarity Analysis
There are two functions in our system that relies on the understanding of scenes in the photos.They are the identification of duplicate images and querying by the scene in images.We would like that the similarity measure to better reflect human perception to the photos.Therefore, a number of different features are employed to evaluate the similarity of photo contents.We used three popular metrics of global and local image features including gist, hog and BoW by SIFT.Our system will try to normalize the these features based on empirical weightings.As a result, images that are similar will be placed closer together in the sorting and searching results.

Face Detection and Recognition
The face detection and recognition module involves the checking of available faces in a photo, then followed by the recognition of similar faces that is the recognition.As illustrated in Figure 3, the detection of faces is performed repetitively to the photo to seek all available human faces.Figure 3(c) illustrated a result from detection, most of the faces can be detected and extracted (on r.h.s.), while noise exits(the plain color region).Then, these identified face regions are forwarded for recognition.We adopted the approach used in an open source biometric recognition package OpenBR [14] for the face recognition.Our processing steps include first to normalize the image to eliminate illumination issues, followed by feature extraction and finally a clustering to group similar faces.However, as a photo may contain multiple faces, there can be case that photos are associated with more than a same face.As our sorting and searching is constrained to be working on the photos in the personal collection which are stored on the cloud.The above process can be performed in an offline manner.The association between faces and photos, or faces to faces are stored in the database for fast retrieval from the mobile client.

Results and Discussion
We have tested various modules in our system individually.Figures 4 and 5 are the screen capture demonstrating the results of different operations.The sort by face and location can present reasonable ordering and grouping as expected.Similar faces or locations are rearranged closer together, so that one can easily choose their desired type of photo without scrolling and seeking from beginning of the collection till the end.However, problem may appear when sorting by face if there are multiple faces in the photo.Our current implementation will try to use the face that occupied the largest space inside the photo, and group this photo based on this largest face.Thus these photos will only appear once in the collection.For photos without faces, they will be placed at the end of the collection.
The search by similarity and face also show satisfactory results, as most of the photos in collection that are similar to the query are included in the query result though it usually contain little of noise.Invited users group, which contain about 4 subjects, finds the provided functionality and corresponding results useful and attractive.

Conclusion
We have demonstrate the use of existing technologies in organizing the daily expending photo collection.By introducing various sorting and searching capabilities, users can reorganize or seek their photos based on the needs.This relieves the tedious and painful manual album creation.Our current prototype focusing on the use of geographical analysis, image similarity analysis and face recognition to facilitate these processes.By leveraging the computation power of cloud, our system enables the use of mobile devices alone in managing user's own photo collection.In future, we are going to investigate the possibility of machine learning to provide customized and personalized photo organizing features.

Figure 1 .
Figure 1.Overview of our system.
(a) Main (b) Image viewer (c) Menu of sorting (d) Menu of searching

Figure 2 .
Figure 2. User inferface of our photo organizing mobile application.
(a) Face detection (b) Face recognition (c) Face detection result

Figure 3 .
Figure 3. Framework for face detection and recognition and some example result of face detection.
(a) Sort by Face (b) Sort by Location (c) Search similar scene (the query image) (d) Search result of similar scene

Figure 4 .
Figure 4. Result of sorting and searching based on similarity.
(a) Search similar face (the query image) (b) Search result of similar faces

Figure 5 .
Figure 5. Result of searching based on faces.