batching & streaming for the brain
in the world of data engineering, there are 2 core workflows for ingesting and processing data.
the first is streaming - you are grabbing and/or receiving data as it's happening and/or being created in the real world. you process and store this data somewhere or using it in an application right away.
the second is batching - you process a large amount of historical data up until a specific moment in time. you process and store this data on some schedule and mainly use it for analytics and sometimes applications.
drawing a parallel to human experience, our conscious interaction with the world mirrors the streaming model. our senses are continuously collecting data from our environment, and our brain processes this information in real time to help us contextualize this data. on the other hand, our unconscious mind is a batch processor. period of rest/downtime like sleep or meditation creates space for the brain to sort through previous experiences, make sense of them, and form new connections.
downtime is crucial for memory consolidation and problem-solving, and staying present & focusing on the current moment gives your unconscious brain the best data to process.