Chantzialexiou, Georgios. Developing messaging and real time processing system for cloud connected cars. Retrieved from https://doi.org/doi:10.7282/t3-b2j5-6490
DescriptionIn the recent years, the interest in developing self driving cars, autonomous drones and connected cars skyrockets. That is leading to the need to develop a cloud messaging system with close to real time capabilities that enable vehicles share information to each other in order to help them improve their navigation. Although, there are many popular existing cloud messaging and processing solutions, these engines introduce diverse characteristics and runtime architectures, so there is a need to analyze not only the resources they require but also the execution time they manage to achieve. The complexity of the task, is also affected by execution parameters of the underlying algorithm. The outcome of such analysis will provide us with the means to understand the advantages and disadvantages of every execution engine under specific circumstances, and also let us deploy user policies in cloud environments that relate to the cost and the time restraints of the executions. For this purpose, we must conduct an experimental analysis on those engines through a profiling process, where we will measure the usage of the resources as well as the overall execution time. The results of this process will enable us to construct static predictive models that could simulate the performance of the engines for varying execution parameters.
In this thesis we used Kafka as our distributed messaging system and measured the communication between cars. We choose Kafka instead of other messaging systems due to its reliability, scalability, ease of use, proven success, and popularity across the big data community.
Furthermore, we used Apache Spark as the real-time processing engine. We chose Spark because it is easy to integrate it with Kafka, for its scalability, reliability, ease of use and its popularity. Moreover, the Machine Learning library of Spark is widely used. In order to analyze the suitability of the above system we developed mini applications that simulate real-world scenarios to analyze the performance of the system. We run experiments using different settings and different workloads and measure performance that help us understand the behavior of the system.