Physics Derivation Graph navigation Sign in

SQL vs CSV vs PKL for data storage

Published 2019-12-12T22:43:00Z by Physics Derivation Graph

The CSV files are sufficient to build a proof of concept. Given that sufficiency, and my lack of familiarity with SQL, perhaps I shouldn't focus on using sqlite for the MVP.
(See https://physicsderivationgraph.blogspot.com/2019/12/mvp-for-pdg-with-sql.html)

CSVs feel hacky and don't enable enforcement of consistency checks. However, they are easier to view and decrease the dependencies.

The easiest solution might be using Pickle (PKL) files rather than converting from Python variables in-memory to a distinct off-line representation. A Pickle file is enables consolidation.