- Details
- Written by: SMAIT
- Category: smart inspect
- Hits: 253
Why do we need smart inspection?
IT system environment protection is the most concerned issue for IT system maintenance personnel. It is related to the normal operation of IT application systems and fault diagnosis and troubleshooting. The best traditional way to protect it is real-time system operation and maintenance, which requires the establishment of an operation and maintenance server and monitoring by professional operation and maintenance personnel. That is, professional operation and maintenance personnel can login to the target system at any time according to the alarm information received to solve possible problems. However, many users do not have such conditions. Therefore, the traditional approach is to sign a regular maintenance contract and conduct periodic inspections of the target system. In this case, hidden dangers between two inspections cannot be discovered in time.
In addition, regular maintenance contracts are usually signed on an annual basis. If the maintenance service provider is different, the maintenance inspection style may be different each year, not to mention the use of historical inspection data. How to meet the requirements of maintenance inspection and keep up with the pace of real-time operation and maintenance as much as possible, smart inspection came into being, which can generate IT system environment operation and maintenance reports every hour for relevant personnel to analyze, so as to take timely measures to optimize the system or solve potential faults, and can generate usage history tracking data.
Smart inspection vs real-time operation and maintenance vs regular maintenance
| Item | Smart Inspection | Realtime operating | Periodical inspection |
| Realtime monitor | No, but daemon running all the time, sampling period from 10s across each hour | Monitor interface , alert records | None, needs more time on digging data |
| Timeliness | Per hour | Instant status / historical status | Every N months |
| content | Comprehensive, host, storage, network，database, disaster backup, application etc | All monitoring facilities instant availability | Depends on operating personnel, generally tend to database |
| Solve problem | Needs experts on-site or configure smart solution for specific problem | Solve immediately | Needs expert goes on-site |
| Report | Per hour | Yearly or half year | Every N months |
| Historical data | Choose to use, e.g. latest data for past several days | Choose to check | Depends on data digging |
| optimize | Should focus on hourly generated report and keep optimizing | SQL performance monitoring | executed as needed, maybe no optimization history record |
| Total cost | low | High | moderate |
Features of Smart Inspection System
The smart inspection system is designed to fill the gap between real-time operation and maintenance and regular maintenance inspections, and can serve as a comprehensive snapshot of the system environment for both to keep for future reference.
Compared with manual regular maintenance inspections, smart inspections have moderate cost advantages, excellent consistency, and shorter inspection cycles. Currently, inspections are generally performed once an hour. Due to the realization of parallelism, the inspection efficiency is greatly improved, that is, the time consumed for inspecting 100 hosts only depends on the slowest machine among them.
Compared with real-time operation and maintenance, smart inspection has more detailed information display and automatically generates electronic documents including md format, docx format, and html format. The md document can be used for historical comparison and archive purpose; and the time period can be set to send emails to relevant personnel..
The smart inspection system can also use various detection methods of real-time operation and maintenance to obtain instant status information, such as database log errors, oracle RAC status, WEBLOGIC application server status, etc.
Utilizing manual inspection and industry rules to quantify and quantify the detection information, and comprehensively form inspection conclusions, effectively avoiding the shortcomings of both.
Highlights of the Smart Inspection System：
Utilizing the historical inspection data generated every hour and presenting them in the latest inspection report, such as the system swap partition space change graph, key file system size change graph, database ASM storage space change graph, network statistics information main error indicators, generate incremental data lists every hour, minute and hour network traffic graphs, and determine whether to display the picture based on the mean square error calculation results of sampled data.
Asynchronous mining of database archive logs, 24-hour interval statistics, table operation count generation, specified table operation tracking; SQL statement bottleneck analysis; Goldengate/Dataguard disaster backup status, process information, process parameters, OGG discarded file error information;
Generating hourly reports for the past week, including database session locks and object locks;
Oracle database optimization suggestions, and enable, disable or intelligently process optimization scripts that affect docx file generation;
Integrate smart password patent technology to achieve effective password management and ensure the security of applications within the system and the normal operation of system services .
Features of smart inspection target documents
The generation of DOCX electronic documents by the smart inspection system is an important part. The electronic documents contain four parts: system overview, inspection summary, text analysis, and attachments. Therefore, the overall implementation plan is to identify the summary information, core information, and supplementary information and enter them into the system overview, text, and attachments respectively, and the comprehensive inspection conclusions into the inspection summary.In addition, the space after the inspection summary can be left blank for manual annotations by system engineers, user feedback, etc. The order of the chapters in the text can be adjusted as needed. The target host can be set with an alias to conform to daily naming habits. In addition, the overlong file system and database table space paths have automatic segmentation processing to improve the readability of the generated document. The target document (docx, html, md) can be automatically sent to relevant personnel as an email attachment according to the configured inspection time and route. Docx documents can embed pictures according to the content for easy viewing.
By embedding post-references, similar tables can be placed adjacent to each other for easy cross-viewing, such as dual-machine load in database cluster RAC;
By setting each category flag, historical status reference query can be achieved, such as inspection reports of database locks;
System summary, all statuses can be clicked to link to detailed information
Smart inspection system structure
The smart inspection system is deployed at the user site who needs to be inspected. It consists of a master control program, configuration files, detection agents, and conversion generation. The configuration files are divided into meta-configuration files and user configuration files. Different users generally have only different user configuration files, and the meta-configuration files are the same. This design greatly simplifies deployment and upgrades.。
The master control program runs on a Linux machine (hereinafter referred to as the inspection machine) and is programmed using Linux bash. Therefore, there are requirements for its version, at least bash4.2 is required (bash3 can perform detection but requires a different machine to generate electronic documents). You can use bash -version to check. The master control program automatically identifies the configuration of generating electronic documents on a different machine through the configuration file. The inspection machine can reach all target machines, and after trust configuration, it can login remotely and execute commands through SSH. It has automatic version management function, generating the difference between the inspection main program and the inspection agent program and their respective historical versions. The master control program generates the background inspection process of each host according to the user-customized configuration file by parallel inspecting , and detects the completion status of each inspection process. Finally, it merges the inspection reports generated by each and converts them into target documents, and sends the inspection reports to the designated personnel according to the pre-set email sending configuration;
The detection agent runs on each target machine and executes various detection commands issued by the master control program. It has Linux/AIX compatibility, realizes universal detection to the greatest extent possible, reduces modifications, and uses lower-level intermediate commands for particularly personalized detection, thereby avoiding frequent modifications of the detection agent due to specific differences in the inspection environment. At the same time, the detection agent can be automatically distributed and updated by the master control program according to the version, and has the characteristics of automatic version management;
The meta-configuration file of the configuration file includes various control parameters, translation mapping, and overall control system operation; the user configuration file includes user-specific configurations, such as inspection target machine strings, etc., but their definition styles are consistent, which brings convenience to transplantation. The user configuration file takes precedence over the meta-configuration file.
The conversion program determines the syntax of the output file of the master control program. Currently, the relatively simple markdown is used, and LaTex can also be used to achieve more complex goals. The conversion program generates idioms for pandoc with reference files to control the style of the generated document. Therefore, the final interactive document should be converted to pdf to avoid requesting reference files when opening. In addition, it should be noted that due to the differences in the processing of tables by various word processing software, the reference file is customizable.
The control logic determines how the inspection content enters into the desired document part. Part of it enters into the system overview based on the system configuration, and part of it makes a reasonable judgment based on the inspection content whether to enter the main text or attachment. In addition, the final generated chapter order can be adjusted through the chapter order.
The startup program can be executed manually or automatically in a scheduled task. It first calls the main control program to generate an md text file (merged from separate md file of each machine), then calls PANDOC to convert it into a docx file and an html file. Both types of files are then archived and compressed. If necessary, it can be configured to automatically send an email to relevant personnel. The startup program can be adjusted according to the bash path. The startup program is powered with parallel control of the main control program, greatly improving the inspection efficiency.
Smart inspection document reader
All personnel related to the IT system environment can find content of interest in the smart inspection system documentation.
Contents reflected by smart inspection
Demonstrate team capabilities, including document design capabilities, system architecture design capabilities, business systems and operations, database design capabilities, application design capabilities, system maintenance capabilities, network storage backup and disaster recovery comprehensive operations and maintenance levels, etc.
