开源实时大数据流挖掘---SAMOA
官方非常赞介绍性 ppt:?http://yahoo.github.io/samoa/ 非官方优秀博客:?http://www.otnira.com/category/projects/ Introducing SAMOA,an open source platform for mining big data streams.https://github.com/yahoo/samoa Machine learning and data mining are well established techniques in the world of IT and especially among web companies and startups. Spam detection,personalization and recommendations are just a few of the applications made possible by mining the huge quantity of data available nowadays. However,“big data” is not only about Volume,but also about Velocity (and Variety,?3V of big data). The usual pipeline for modeling data (what “data scientists” do) involves taking a sample from production data,cleaning and preprocessing it to make it usable,training a model for the task at hand and finally deploying it to production. The final output of this process is a pipeline that needs to run periodically (and be maintained) in order to keep the model up to date. Hadoop and its ecosystem (e.g.,?Mahout) have proven to be an extremely successful platform to support this process at web scale. However,no solution is perfect and big data is?"data whose characteristics forces us to look beyond the traditional methods that are prevalent at the time". The current challenge is to move towards analyzing data as soon as it arrives into the system,nearly in real-time. For example,models for mail spam detection get outdated with time and need to be retrained with new data. New data (i.e.,spam reports) comes in continuously and the model starts being outdated the moment it is deployed: all the new data is sitting without creating any value until the next model update. On the contrary,incorporating new data as soon as it arrives is what the “Velocity” in big data is about.?In this case,Hadoop is not the ideal tool to cope with streams of fast changing data. Distributed stream processing engines are emerging as the platform of choice to handle this use case. Examples of these platforms are?Storm,?S4,and recently?Samza. These platforms join the scalability of distributed processing with the fast response of stream processing. Yahoo has already adopted Storm as a key technology for low-latency big data processing. Alas,currently there is no common solution for mining big data streams,that is,for doing machine learning on streams on a distributed environment. Enter SAMOASAMOA (Scalable Advanced Massive Online Analysis) is a framework for mining big data streams. As most of the big data ecosystem,it is written in Java. It features a pluggable architecture that allows it to run on several distributed stream processing engines such as Storm and S4. SAMOA includes distributed algorithms for the most common machine learning tasks such as classification and clustering. For a simple analogy,you can think of SAMOA as Mahout for streaming. SAMOA is both a platform and a library. As a platform,it allows the algorithm developer to abstract from the underlying execution engine,and therefore reuse their code to run on different engines. It also allows to easily write plug-in modules to port SAMOA to different execution engines. As a library,SAMOA contains state-of-the-art implementations of algorithms for distributed machine learning on streams. The first alpha release allows classification and clustering. For classification,we implemented a Vertical Hoeffding Tree (VHT),a distributed streaming version of decision trees tailored for sparse data (e.g.,text). For clustering,we included a distributed algorithm based on CluStream. The library also includes meta-algorithms such as bagging. HOW DOES IT WORK?An algorithm in SAMOA is represented by a series of nodes communicating via messages along streams that connect pairs of nodes (a graph). Borrowing the terminology from Storm,this is called a Topology. Each node in the Topology is a Processor that sends messages to a Stream. The user code that implements the algorithm resides inside a Processor. Figure 3 shows an example of a Processor joining two stream from two source Processors. Here is a code snippet to build such a topology in SAMOA. TopologyBuilder builder; Processor sourceOne = newSourceProcessor();.addProcessor(sourceOne);Stream streamOne createStream sourceTwo sourceTwo streamTwo join JoinProcessorjoin).connectInputShufflestreamOneconnectInputKeystreamTwo);
|