可观测性十大场景 | 关于保险行业开门红期间应用性能的端到端全栈可观测

2024-03-07

WechatIMG4239-1709798795250

【场景概述】

保险行业的“开门红”是每年10月份到次年2月份的业绩冲刺期,各大保险公司纷纷推出独具特色的理财产品,吸引广大客户的目光,以期在新年伊始便赢得“开门红”的吉祥兆头。这段时期,保险收入占比接近全年收入的50%,总体收入超过万亿元,也被称为保险行业的“双十一”。

在这盛大的促销活动中,单日最大保单量能突破3亿,平均每秒处理交易数高达3.2万笔。为确保系统的稳定运行,特别是在首爆日、1月1日等重点活动日,运维团队必须24小时待命,实时监控系统运行状态、性能指标和错误日志,及时发现并处理潜在问题和异常情况。面对高峰时段突发状况的挑战,运维人员需要快速看到相应的运行数据,根据分析结果做出故障处理,保障系统的稳定性与可靠性,这对运维人员的专业技能与应对能力提出了极高的要求。

【发展挑战】

1、促销活动开始后,一旦网站出现性能问题,导致页面加载缓慢、订单处理延迟等严重影响用户体验的情况,由于缺乏对业务质量的量化评估,运维团队难以快速定位问题根源。

2、当应用端发生多起用户账户异常操作事件,由于缺乏有效的追踪机制和日志记录,运维团队只能通过用户投诉来发现问题,无法主动了解用户在整个交易过程中的体验情况。

3、新产品上线后频繁出现系统崩溃、服务不可用等故障情况,由于缺乏清晰的职责划分和考核机制,导致故障诊断流程变得复杂且效率低下,并且职责边界模糊不清。

【服务内容】

博睿数据通过一体化智能可观测平台Bonree ONE,构建端到端全栈可观测场景,涵盖业务部门、运维团队以及前后端开发的全流程。平台能够实时洞察应用登录、投保流程、双录等关键业务接口的性能表现,通过精确计算接口响应时间、错误率等关键指标,实时评估接口健康状态,确保问题能够被及时发现并解决。此外,通过一键追踪到代码,支持运维团队对代码栈、错误、日志等细节的深入查看。同时,调用链能够关联至基础架构信息,包括CPU、内存、磁盘等主机详情,结合仪表盘的主动发现功能和告警的被动接收机制,迅速响应客户投诉,精准诊断问题,并实现快速恢复,从而实现“开门红”期间 0 故障 0 投诉,助力企业做到真正的开门红。最后,博睿数据总结出五大黄金指标,为保险行业搭建一套可复制推广的可观测性体系。

【服务亮点】

“四看原则”实现对业务质量的全面监控与管理

查看业务接口的性能。通过业务接口的响应时间、错误率等计算当前接口健康评分,评估接口健康状态,以便及时发现并解决问题。

01

查看应用整体系统的综合性能。如响应时间、错误率、cpu、磁盘、负载等主机与数据库性能情况,能够对整体系统进行监控和管理。

02

查看应用前端相关的性能。如整体用户访问情况、崩溃、卡顿、页面与视图性能等,以优化用户体验和页面性能。

03

查看应用后端相关的性能。如服务调用耗时、数据库查询时间、第三方调用时间等,以优化系统运行效率和稳定性。

04

双重故障诊断策略加固稳定性

通过仪表盘主动发现与告警被动接收两种方式结合,能够快速的进行故障诊断。当指标出现异常时,可通过仪表盘颜色变化提示相关人员,并提供详细的数据和图表展示,使相关人员能够迅速点击查看相关调用链,深入分析原因,精准定位问题根源,主动介入避免潜在问题发生。一旦出现突发事件,告警机制能够及时捕捉问题,通过根因分析与故障回放,确保问题在第一时间内被发现,为后续快速修复奠定基础。

05

五大黄金指标精准护航保险行业

经过对保险行业客户“开门红”期间的长期服务的深入研究,博睿数据针对用户端、应用层、网络层、资源层、中间件、数据库和业务端七大层面,从超过2000个预置指标中精心筛选出五个关键性指标及其阈值:请求错误率、很慢调用占比、数据库错误率、崩溃率和接口错误率。基于这些指标,博睿数据为保险行业搭建了一套可复制推广的端到端全栈可观测性体系,通过数据源管理与拓扑监测,一旦监控指标出现异常,系统将自动生成告警,通知运维团队迅速介入。运维团队可对系统日志、数据库查询、网络连接等进行检查,及时发现并解决潜在问题,确保系统正常运行,为用户提供卓越的服务体验。

图片6

文章标签

可观测性

相关文章

FreeMarker template error (DEBUG mode; use RETHROW in production!): The string doesn't match the expected date/time/date-time format. The string to parse was: "2024年3月5日". The expected format was: "MMM d, y". The nested reason given follows: Unparseable date: "2024年3月5日" ---- FTL stack trace ("~" means nesting-related): - Failed at: #if "2024年3月5日"?date lt item.createTi... [in template "themes/halo_quickstarter/post_newsDetail.ftl" at line 106, column 25] ---- Java stack trace (for programmers): ---- freemarker.core._TemplateModelException: [... Exception message was already printed; see it above ...] at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.parse(BuiltInsForMultipleTypes.java:220) at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.getAsDateModel(BuiltInsForMultipleTypes.java:190) at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.getAsDate(BuiltInsForMultipleTypes.java:197) at freemarker.core.EvalUtil.modelToDate(EvalUtil.java:86) at freemarker.core.EvalUtil.compare(EvalUtil.java:270) at freemarker.core.EvalUtil.compare(EvalUtil.java:115) at freemarker.core.ComparisonExpression.evalToBoolean(ComparisonExpression.java:78) at freemarker.core.IfBlock.accept(IfBlock.java:49) at freemarker.core.Environment.visit(Environment.java:370) at freemarker.core.IteratorBlock$IterationContext.executedNestedContentForCollOrSeqListing(IteratorBlock.java:321) at freemarker.core.IteratorBlock$IterationContext.executeNestedContent(IteratorBlock.java:271) at freemarker.core.IteratorBlock$IterationContext.accept(IteratorBlock.java:244) at freemarker.core.Environment.visitIteratorBlock(Environment.java:644) at freemarker.core.IteratorBlock.acceptWithResult(IteratorBlock.java:108) at freemarker.core.IteratorBlock.accept(IteratorBlock.java:94) at freemarker.core.Environment.visit(Environment.java:334) at freemarker.core.Environment.visit(Environment.java:340) at freemarker.core.Environment.visit(Environment.java:340) at freemarker.core.Environment.process(Environment.java:313) at freemarker.template.Template.process(Template.java:383) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:391) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.doRender(FreeMarkerView.java:304) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.renderMergedTemplateModel(FreeMarkerView.java:255) at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179) at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1401) at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1145) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1084) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:497) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:230) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.cas.UserInfoFilter.doFilter(UserInfoFilter.java:37) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.jasig.cas.client.util.AssertionThreadLocalFilter.doFilter(AssertionThreadLocalFilter.java:54) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:75) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.cas.MyAbstractTicketValidationFilter.doFilter(MyAbstractTicketValidationFilter.java:215) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.cas.MyAuthenticationNoLoginFilter.doFilter(MyAuthenticationNoLoginFilter.java:57) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.security.filter.ContentFilter.doAuthenticate(ContentFilter.java:69) at run.halo.app.security.filter.AbstractAuthenticationFilter.doFilterInternal(AbstractAuthenticationFilter.java:229) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.filter.CorsFilter.doFilter(CorsFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at run.halo.app.filter.LogFilter.doFilterInternal(LogFilter.java:40) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: freemarker.core.UnparsableValueException: Unparseable date: "2024年3月5日" at freemarker.core.JavaTemplateDateFormat.parse(JavaTemplateDateFormat.java:51) at freemarker.core.JavaTemplateDateFormat.parse(JavaTemplateDateFormat.java:33) at freemarker.core.BuiltInsForMultipleTypes$dateBI$DateParser.parse(BuiltInsForMultipleTypes.java:213) ... 118 more Caused by: java.text.ParseException: Unparseable date: "2024年3月5日" at java.base/java.text.DateFormat.parse(DateFormat.java:395) at freemarker.core.JavaTemplateDateFormat.parse(JavaTemplateDateFormat.java:49) ... 120 more