Data ingestion layer - ingest for processing and storage

A fast ingestion layer is one of the key layers in the Lambda Architecture pattern. This layer needs to control how fast data can be delivered into the working models of the Lambda Architecture. Some of the key specifications of this layer are:

• It must be highly scalable with on-demand scalability to be able to scale based on varying load conditions
• It must be fault tolerant with both fail-safety (recovery) as well as fail-over (resiliency)
• This layer must be able to support multi-thread and multi-event execution
• This layer must be able to quickly transform the acquired data structure into the target data formats as needed by the processing layers of the Lambda Architecture
• This layer must ensure that all of the data delivered is in its purest form for further processing