Is temporal database decentralized storage
the solution is an efficient insert query for massive data
it is applied to the large-scale data statistical analysis of the Internet and the information collection of the Internet of things
time point is very important for temporal database, and high throughput determines its value.
Siji yourong time series database is an innovative big data processing proct launched by Puhua in the face of this fast-growing big data market and technical challenges of the Internet of things. It does not rely on any third-party software, nor does it optimize or package an open source database or stream computing proct. Instead, it is absorbing many traditional relational databases, NoSQL databases, etc After the advantages of streaming computing engine, message queue and other software, the self-developed procts have their own unique advantages in temporal space big data processing
< UL >· more than 10 times performance improvement: an innovative data storage structure is defined. A single core can process at least 20000 requests per second, insert millions of data points and read more than 10 million data points, which is more than 10 times faster than the existing general database
· the cost of hardware or cloud services is reced to 1 / 5: e to super performance, the computing resources are less than 1 / 5 of the general big data solution; Through column storage and advanced compression algorithm, the storage space is less than 1 / 10 of the general database
· full stack sequential data processing engine: integrates database, message queue, cache, streaming computing and other functions, so that the application does not need to integrate Kafka / redis / HBase / HDFS and other software, greatly recing the complexity and cost of application development and maintenance
· powerful analysis function: whether it is data ten years ago or one second ago, you can query it within a specified time range. Data can be aggregated on a timeline or on multiple devices. Temporary query can be carried out at any time through shell, python, R and MATLAB
· seamless connection with third-party tools: without one line of code, it can be integrated with telegraf, grafana, MATLAB, R and other tools. In the future, mqtt, OPC and other tools will be supported, and they can also be seamlessly connected with Bi tools
· zero operation and maintenance cost, zero learning cost: installation and cluster can be done in one second, no need to separate database and table, real-time backup. Support standard SQL statements, support JDBC, restful connection, support Python / Java / C / C + + / go and other development languages, similar to MySQL, zero learning cost
with Siji yourong time series database, the overall cost of typical big data platforms of Internet of things, Internet of vehicles and instrial Internet can be reced to 1 / 5 of the existing cost. With the same hardware resources, sijiyourong sequential database can increase the processing capacity and capacity of the system by more than five times
at the same time, compared with HBase and other databases, using PWS extremely tolerant sequential database for storage has the following advantages:
1. Storage space is greatly saved, less than 1 / 10 of HBase
2. Server resources are greatly saved, Estimated less than 1 / 5
3. Query speed increased by at least 10 times
4. Provide remote disaster recovery backup scheme
5. Support ad hoc query through standard SQL
6. Data over retention time, automatically delete
7. Zero management, installation, deployment, maintenance is extremely simple, one click to do
At present, the data of time series database of network cloud can be backed up to the object storage bos of network cloud
of course, any temporal database can be backed up to the object storage
object storage mainly provides stable, safe, efficient and highly scalable cloud storage services. Any amount and form of unstructured data can be stored in the BOS, and the data can be managed and processed. BOS supports standard, low frequency, cold storage and other storage types to meet the storage requirements of all kinds of scenes
In some cases where the amount of data is large and the operation of database is frequent. At this time, you need to store the whole datatable into the database
First, create a new databasedatatable once_ rec_ date = new DataTable();
this database must have the same column position and size as the target database. In particular, type, and location. As like as two peas, the location of the column and the location of the target database are exactly the same. Because it's all block storage, the address and everything must be the same. You can't lose one column. The auto increment column can be empty there
extended data
basic knowledge of database introction:
classification of database
relational database: after mathematical theory verification, it can save all kinds of relational data in real life, and the data stored in the database is based on tables; Non relational database: usually used to solve some specific requirements, such as: data cache, high concurrency access. There are many ways to store data. For example, redis database stores data in the form of key value pairs
create database: create database_ Name
delete database: drop database database_ Name
select database: usedatabase_ Name
create data table: create table_ name (column_ name column_ Type)
delete data table: drop table table_ Name
update data table information:
Add table field: alter table_ name ADD new_ Column datatype
use the first keyword to adjust the order of new columns to the first column of the data table: alter table_ name ADD new_ Column datatype first
use the after keyword to adjust the new column to the specified column of the data table: alter table_ name ADD new_ column DATATYPE AFTER old_ column
The method of saving data is as follows
1, data classification
you should classify the data you want to save effectively for a long time, and divide it into three types: core data, important data and ordinary data. Then, according to the reliability requirements of different levels of data, the corresponding methods are worked out< br />
2 / 6
compressed storage space
the data to be saved may include word, picture, audio, video and other types of files, which can be converted into smaller format, so that we can store more data< br />
3 / 6
select storage media
in fact, there are many ways to save data, such as optical disk, hard disk, mechanical hard disk, network disk, so if you have a certain amount of money, it is recommended that you choose a more secure storage media< br />
4 / 6
check regularly
no matter what media you store, if you want to save data for a long time, you should check it regularly. Maintain it properly, and fix it if there is a vulnerability< br />
<
5 / 6
important data should be backed up
not only one of important data, but at least three copies. When one of them is lost by accident, you will not lose the data completely< br />
6 / 6
put the files directly on the file system
don't make another package file and save a bunch of files after packaging. When the media fails, the package files are difficult to recover, while the files directly on the file system are relatively easy to recover< br />