大数据技术在运维平台系统中的应用

2023-04-05

大数据以及信息化技术的飞速发展,给各行各业建立数字化体系提供了强有力的技术支持,对于运维平台系统来说,大数据技术也直观重要,通过对大数据集成开发、分布式存储以及数据分析及手机能功能,帮助企业建立强大的数据库和可视化的智能数据分析体系,更加直观的进行市场预测与开发。

智能运维平台系统

对于企业来说,通过构建基于国产统一分布式大数据技术的一体化数据中台( OMDB ) ,解决了多维异构大数据技术的统一实现(即用1种单一数据库代替由Hadoop、MongoDB、 Mysql、DB2Oracle、ElasticSearch、Neo4j、 influxDB等混合数据库类型的大数据平台),实现运维大数据集成开发、分布式存储、建模和分析、数据服务等功能,对结构化数据、非结构化数据、关系图数据以及时序数据等大数据进行统一的存储、建模、分析和服务,支持面向事务型和面向分析型等数据处理需求,支持实时及批量方式对外提供数据服务。

技术层面,通过多种数据采集技术实现了对运维、批量、日志、资产、拓扑关系及业务交易等大数据的采集分析、实时同步和智能关联分析,参考E-R数据建模方法,根据运维大数据的环境特点,结合“可拓学”理论进行数据建模方法的理论及实践创新,通过E-A-R(实体、活动、关系)模)型代替传统E-R模型(实体、关系),对不同类型的运维、监控及资产等数据(时序数据、日志数据、表数据、拓扑图数据、文档数据等)进行统一建模。通过对时序、关键词、自然语言等进行数据检索,采用OMDB作为中台数据持久化层,利用分布式、扩展性强、多中心多活、无中心节点的特性,实现不同类型数据的统一存储.同时采用事件处理(连锁)理论(人->流程、物->资源、事->操作)开发了统--调度引擎,实现运维监、管、控等的统--原子编排调度。

业务层面,通过面向交易业务代码级别、终端客户体验级别实现端到端交易的全链路追踪监控,面向不同角色构建了十几种银行业务及系统性能视图,为业务部门产品营销、跟踪、分析提供了强大的辅助决策支持能力。同时基于大数据平台的智能运维系统构建了版本投产与年终决算两套管理类指挥视图,实现了数字化、可视化的实时运营,有力支撑了科技管理能力提升。

实践证明,大数据技术在智能运维平台系统中的应用,简化了数据架构、技术架构等,降低了系统运维难度和复杂度,使数据管理更加系统,数据使用更加便捷,提升了数据分析能力、服务能力及运营能力,同时大大提高了科技的管理能力、决策能力以及客户满意度。

文章标签

智能运维 一体化智能可观测

相关文章

FreeMarker template error (DEBUG mode; use RETHROW in production!): The string doesn't match the expected date/time/date-time format. The string to parse was: "2024年3月5日". The expected format was: "MMM d, y". The nested reason given follows: Unparseable date: "2024年3月5日" ---- FTL stack trace ("~" means nesting-related): - Failed at: #if "2024年3月5日"?date lt item.createTi... [in template "themes/halo_quickstarter/post_newsDetail.ftl" at line 106, column 25] ---- Java stack trace (for programmers): ---- freemarker.core._TemplateModelException: [... Exception message was already printed; see it above ...] at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.parse(BuiltInsForMultipleTypes.java:220) at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.getAsDateModel(BuiltInsForMultipleTypes.java:190) at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.getAsDate(BuiltInsForMultipleTypes.java:197) at freemarker.core.EvalUtil.modelToDate(EvalUtil.java:86) at freemarker.core.EvalUtil.compare(EvalUtil.java:270) at freemarker.core.EvalUtil.compare(EvalUtil.java:115) at freemarker.core.ComparisonExpression.evalToBoolean(ComparisonExpression.java:78) at freemarker.core.IfBlock.accept(IfBlock.java:49) at freemarker.core.Environment.visit(Environment.java:370) at freemarker.core.IteratorBlock$IterationContext.executedNestedContentForCollOrSeqListing(IteratorBlock.java:321) at freemarker.core.IteratorBlock$IterationContext.executeNestedContent(IteratorBlock.java:271) at freemarker.core.IteratorBlock$IterationContext.accept(IteratorBlock.java:244) at freemarker.core.Environment.visitIteratorBlock(Environment.java:644) at freemarker.core.IteratorBlock.acceptWithResult(IteratorBlock.java:108) at freemarker.core.IteratorBlock.accept(IteratorBlock.java:94) at freemarker.core.Environment.visit(Environment.java:334) at freemarker.core.Environment.visit(Environment.java:340) at freemarker.core.Environment.visit(Environment.java:340) at freemarker.core.Environment.process(Environment.java:313) at freemarker.template.Template.process(Template.java:383) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:391) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.doRender(FreeMarkerView.java:304) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.renderMergedTemplateModel(FreeMarkerView.java:255) at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179) at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1401) at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1145) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1084) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:497) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:230) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.cas.UserInfoFilter.doFilter(UserInfoFilter.java:37) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.jasig.cas.client.util.AssertionThreadLocalFilter.doFilter(AssertionThreadLocalFilter.java:54) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:75) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.cas.MyAbstractTicketValidationFilter.doFilter(MyAbstractTicketValidationFilter.java:215) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.cas.MyAuthenticationNoLoginFilter.doFilter(MyAuthenticationNoLoginFilter.java:57) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.security.filter.ContentFilter.doAuthenticate(ContentFilter.java:69) at run.halo.app.security.filter.AbstractAuthenticationFilter.doFilterInternal(AbstractAuthenticationFilter.java:229) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.filter.CorsFilter.doFilter(CorsFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.filter.LogFilter.doFilterInternal(LogFilter.java:40) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: freemarker.core.UnparsableValueException: Unparseable date: "2024年3月5日" at freemarker.core.JavaTemplateDateFormat.parse(JavaTemplateDateFormat.java:51) at freemarker.core.JavaTemplateDateFormat.parse(JavaTemplateDateFormat.java:33) at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.parse(BuiltInsForMultipleTypes.java:213) ... 118 more Caused by: java.text.ParseException: Unparseable date: "2024年3月5日" at java.base/java.text.DateFormat.parse(DateFormat.java:395) at freemarker.core.JavaTemplateDateFormat.parse(JavaTemplateDateFormat.java:49) ... 120 more