What challenges does machine learning face in the field of data?

The importance of data for machine learning is well known. Understanding data access patterns will help data scientists determine the right storage infrastructure for their projects. Data infrastructure makes machine learning possible. However, once it is used, machine learning faces key data challenges, which need to be solved first: integrity, sparsity and quality. 1. Integrity Data integrity is the guarantee of data accuracy and consistency. The data chain of custody is essential to prove that data is not compromised as it moves through pipelines and locations. When data capture and ingestion are under control, you can verify its integrity relatively easily. However, when working with others, it is difficult to verify. When generating data, there was no security certificate for external data. You can't ensure that the data record is exactly what you expected, or that the data received is exactly the same as the original record. There are some interesting concepts about IOT data and blockchain, but before this concept is widely adopted, data integrity depends on the combination of security technology and strategy. For example, since the data may be threatened during static or transmission, the data transmitted through the network should use HTTPS and be encrypted when static. On the other hand, access control should be policy driven to avoid human errors. 2. Sparsity In this case, sparsity applies to metadata. Generally, metadata fields are incomplete. Some fields have been filled in and some fields are left blank. If the data is generated from a single source, it may be due to human lack of norms or knowledge. However, if the data comes from various sources without a standard definition of metadata, each dataset may have completely different fields. Therefore, when they are combined, the completed fields may not correspond. Currently, there is no industry standard for what metadata to capture. However, metadata is as important as the data itself. How do you associate and filter data when you have the same type of data populated with different metadata fields? If you take a buoy as an example, the initial data sensor collects the water temperature every ten minutes, while the newer buoy collects the water temperature every three minutes. The only way to associate data is to expose it at capture time through metadata. When scientists do historical analysis, they need metadata so that they can adjust their models accordingly. 3. Quality Many data scientists want to use data from external sources. However, there is usually no quality control or assurance on how to capture raw data. Do you believe in the accuracy of external data? This is a good example. Sensors on buoys floating in the ocean collect data about ocean temperature. However, when the sensor cannot collect the temperature, it will record 999. In addition, before 2000, only two figures were used to record the number of years. However, after 2000, the number recorded changed to four. Therefore, we need to understand the quality of data and how to prepare data. In this case, scientists analyzing buoy data can use average, mean, minimum and maximum to visualize the original data, capture these database errors and clean them up accordingly. Secure data collaboration If your industry needs to constantly exchange data with external organizations, it is best to open the source code of your data and meta format, because these standards are broader than many proprietary standards. Even better, you can launch an industry open standards committee to allow others to participate and contribute. A good example is "open goal", a "public-private partnership for systematic drug target identification and prioritization using human genetics and genomics data." In particular, the research on the data ecosystem has become highly complex. Partners inside and outside the organization need to quickly access data and simplify data management. Machine learning has many challenges. The first step is to start the project with the correct data and infrastructure. How to start? Data quality, sparsity and integrity directly affect the accuracy of the final model and are some of the biggest challenges facing machine learning today. Organizations with clear data definitions, policies and exploring industry-specific data standards will benefit from short-term and long-term projects. If you haven't, your organization should first define its own data collection policy and metadata format, and then apply standard security technology. Data quality and sparsity go hand in hand. Next, set metadata policy and ensure that the captured qualitative data can be used to verify the effectiveness of data. Finally, in order to ensure data integrity, you can generate data When applying digital certificates, SSL should be enforced during transmission and encryption should always be enabled.