Hi,
Is it possible to use Zarr library (chunked, compressed, N-dimensional arrays in Python) for large scale potential field inversion? Are there any examples?
Hi @Alexey,
Wowzer, that’s a late reply. I rarely come here.
Yes you got it! We started using dask+zarr a while back, but it has only been officially added recently with the 0.14.0 release. You can turn the parallelization (that uses zarr) by adding this to your imports:
import SimPEG.dask
The beauty of this, as you probably know, is that you don’t need much RAM anymore. Just a fast SSD drive to move chunks of sensitivity to your processors. Dask handles the linear algebra all for us.
1 Like
Thank you for the information. I’ll try.
I saw about dask in the 0.14 release info, but didn’t suspect that there is also zarr.
How big did you try @Alexey ?