KMITL

Permanent URI for this communityhttps://dspace.kmitl.ac.th/handle/123456789/1

Browse

Settings

Access type: UnknownSubject: search.filters.subject.Apache Spark

Search Results

Now showing 1 - 2 of 2
  • Loading...
    Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    A real-time recommendation engine using lambda architecture
    (2018-06-01)
    Numnonda, Thanisa
    In a data science theory, the recommended methodology is one of the most popular theories and has been deployed in many real industries. However, one of the most challenging problems these days is how to recommend items with massively streaming data. Therefore, this paper aims to do a real-time recommendation engine using the Lambda architecture. The Apache Hadoop and Apache Spark frameworks were used in this research to process the MovieLens dataset comprised 100 K and 20 M ratings from the GroupLens research. Using alternating least squares (ALS) and k-means algorithms, the top K recommendation movies and the top K trending movies for each user were shown as results. Additionally, the mean squared error (MSE) and within cluster sum of squared error (WCSS) had been computed to evaluate the performance of the ALS and k-means algorithms, sequentially. The results showed that they are acceptable since the MSE and WCSS values are low when comparing to the size of data. However, they can still be improved by tuning some parameters.
  • Loading...
    Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Adapting K-means clustering to identify spatial patterns in storms
    (2016-01-01)
    Gupta, Upa
    ;
    Jitkajornwanich, Kulsawasd
    ;
    Elmasri, Ramez
    ;
    Fegaras, Leonidas
    This paper extends our previous work on deriving meaningful storm patterns from very large rainfall data. In an earlier work, we described MapReduce-based algorithms to identify three types of the storms: local, hourly and overall storms. In general, local storms have temporal characteristics of the storms at a particular site, hourly storms have spatial characteristics of the storms at a particular hour and overall storms have both spatial and temporal characteristics of the storm. We aim to find meaningful patterns and predict trajectories in the spatio-temporal data (i.e. overall storms which are sets of geographically overlapping, consecutive hourly storms). In this paper, we adapt K-Means clustering to find different types of hourly storms based on their shapes and sizes. Since the rainfall data are typically larger than the memory capacity of a single computer, we have implemented this clustering algorithm in Apache Spark, which is a distributed data processing framework, and have run our experiments on a computer cluster.