A web log real-time analysis platform based on stream computing

Jun Zhang; Li Zhang; Peiyi Tang; Taizhi Lv

doi:10.1117/12.2660112

3 February 2023 A web log real-time analysis platform based on stream computing

Jun Zhang, Li Zhang, Peiyi Tang, Taizhi Lv

Proceedings Volume 12511, Third International Conference on Computer Vision and Data Mining (ICCVDM 2022); 125111W (2023) https://doi.org/10.1117/12.2660112
Event: Third International Conference on Computer Vision and Data Mining (ICCVDM 2022), 2022, Hulun Buir, China

Abstract

With the rapid development of Internet technology, web applications have been widely used. How to ensure the reliability and high performance of web applications has become the focus of web site management. When a web application provides services, huge web logs are generated. These logs contain a great deal of information about users access to this web application. Real-time analysis of web logs can obtain system performance indicators and bottlenecks. In order to improve the reliability and performance of web applications, a real-time web log analysis platform based on stream computing is designed and implemented. The platform collects web log data by Flume, realizes data flow by Kafka message queue, analyzes web log by the Flink stream computing platform, stores the computing results in Redis for real-time query, and in Doris for historical data query. The availability of the platform is proved by running in a real environment, and it can improve the performance and reliability of web applications.

Citation Download Citation

Jun Zhang, Li Zhang, Peiyi Tang, and Taizhi Lv "A web log real-time analysis platform based on stream computing", Proc. SPIE 12511, Third International Conference on Computer Vision and Data Mining (ICCVDM 2022), 125111W (3 February 2023); https://doi.org/10.1117/12.2660112

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available