Predicting diffusive size factors of a rock sample#
This tutorial demonstrates how to use PoreSpy’s diffusive_size_factor_AI and diffusive_size_factor_DNS functions to examine the application of the AI-based size factor model on a real rock sample. Due to the computational cost of the DNS method, only a subsection of the image is used in this examples. However, the method can be applied similarly on the entir image of the rock sample.
This is a computationally intensive example. The full AI/DNS comparison also requires TensorFlow, h5py, scikit-learn, OpenPNM, and the pretrained weights downloaded from PMEAL/sf-model-lib in the setup cells below. Loading the bundled image does not require downloading the full rock scan.
For this specific sample:
Binary image data was obtained from the digital rock portals for Leopard rock sample with Voxel length of 2.25 um.
Import libaries and the AI model path#
import importlib
import os
import time
import warnings
import h5py # if there was error importing, please install the h5py package
import numpy as np
from matplotlib import pyplot as plt
from sklearn.metrics import r2_score
import porespy as ps
ps.visualization.set_mpl_style()
warnings.filterwarnings("ignore")
Ensure the existence of model path, and create one if non-existant:
if not os.path.exists("sf-model-lib"):
!git clone https://github.com/PMEAL/sf-model-lib
Cloning into 'sf-model-lib'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 20% (1/5)
remote: Counting objects: 40% (2/5)
remote: Counting objects: 60% (3/5)
remote: Counting objects: 80% (4/5)
remote: Counting objects: 100% (5/5)
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 20% (1/5)
remote: Compressing objects: 40% (2/5)
remote: Compressing objects: 60% (3/5)
remote: Compressing objects: 80% (4/5)
remote: Compressing objects: 100% (5/5)
remote: Compressing objects: 100% (5/5), done.
Receiving objects: 2% (1/36)
Receiving objects: 5% (2/36)
Receiving objects: 8% (3/36)
Receiving objects: 11% (4/36)
Receiving objects: 13% (5/36)
Receiving objects: 16% (6/36)
Receiving objects: 16% (6/36), 46.66 MiB | 46.59 MiB/s
Receiving objects: 19% (7/36), 69.98 MiB | 46.51 MiB/s
Receiving objects: 19% (7/36), 92.75 MiB | 46.27 MiB/s
Receiving objects: 19% (7/36), 140.67 MiB | 46.82 MiB/s
Receiving objects: 22% (8/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 25% (9/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 27% (10/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 30% (11/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 33% (12/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 36% (13/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 38% (14/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 41% (15/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 44% (16/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 47% (17/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 50% (18/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 52% (19/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 55% (20/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 58% (21/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 61% (22/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 63% (23/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 66% (24/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 69% (25/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 72% (26/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 75% (27/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 77% (28/36), 166.43 MiB | 47.49 MiB/s
Receiving objects: 77% (28/36), 188.35 MiB | 47.04 MiB/s
Receiving objects: 77% (28/36), 230.44 MiB | 48.14 MiB/s
Receiving objects: 80% (29/36), 250.30 MiB | 45.22 MiB/s
Receiving objects: 80% (29/36), 271.67 MiB | 44.82 MiB/s
Receiving objects: 80% (29/36), 307.03 MiB | 42.54 MiB/s
remote: Total 36 (delta 0), reused 2 (delta 0), pack-reused 31 (from 1)
Receiving objects: 83% (30/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 86% (31/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 88% (32/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 91% (33/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 94% (34/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 97% (35/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 100% (36/36), 325.37 MiB | 41.02 MiB/s
Receiving objects: 100% (36/36), 334.57 MiB | 42.94 MiB/s, done.
Resolving deltas: 0% (0/8)
Resolving deltas: 12% (1/8)
Resolving deltas: 25% (2/8)
Resolving deltas: 37% (3/8)
Resolving deltas: 50% (4/8)
Resolving deltas: 62% (5/8)
Resolving deltas: 75% (6/8)
Resolving deltas: 87% (7/8)
Resolving deltas: 100% (8/8)
Resolving deltas: 100% (8/8), done.
Updating files: 75% (6/8)
Updating files: 87% (7/8)
Updating files: 100% (8/8)
Updating files: 100% (8/8), done.
In the cell below we import the proper library and assign the weights:
h5tools = importlib.import_module("sf-model-lib.h5tools")
DIR_WEIGHTS = "sf-model-lib/diffusion"
fname_in = [f"{DIR_WEIGHTS}/model_weights_part{j}.h5" for j in [0, 1]]
# Identifying hdf5 files and merging them together, the result would be a unique file
h5tools.combine(fname_in, fname_out=f"{DIR_WEIGHTS}/model_weights.h5")
Next, we import folder path are related libraries (these chould be installed before usage) and define the AI path for
the training data the wieghts. The training data is used in the backend of the diffusive_size_factor_AI to rescale the predictions:
path_AI = "./sf-model-lib/diffusion"
path_train = os.path.join(path_AI, "g_train_original.hdf5")
path_weights = os.path.join(path_AI, "model_weights.h5")
g_train = h5py.File(path_train, "r")["g_train"][()]
Reading image of the rock sample#
As the image of the rock smaple was large, only a subsection of the image is used in this tutorial for rapid demonstration purposes. We saved a subsection of the Leopard rock sample image of size \(100^3\) in PoreSpy’s test/fixtures folder, which is used for this tutorial. However, the steps to download, read, and slice the rock sample image are provided as markdown cells in the next section for reference.
Clone or download the PoreSpy source repository and run this notebook from within that checkout. The small image fixture is included under test/fixtures; a pip installation alone does not include it. The cell below locates the same fixture from the original notebook folder and the documentation build’s copy. The full raw-image download described below is optional and is not needed to use this fixture.
from pathlib import Path
for repo_root in (Path.cwd(), *Path.cwd().parents):
sample_path = repo_root / "test" / "fixtures" / "image_Leopard_slice100.npy"
if sample_path.is_file():
break
else:
raise FileNotFoundError(
"image_Leopard_slice100.npy was not found. Run this notebook from "
"within a PoreSpy source checkout containing test/fixtures."
)
voxel_size = 2.25e-6
im = np.load(sample_path)
Additional info: steps to create a subsection of the rock sample#
Downloading the image of the rock sample: The cell below creates and ensures the existence of the specific sample path (here rock sample Leopard folder). The image of the sample can be downloaded through a link as of the cell below “url” and calling
download_imagefunction, or added seperately as a file to the rock sample folder. If the file is downloaded separately, the last line of the cell should be commented to prevent re-download.
Note: The downloadable link for a binary image in the digital rocks portal can be found by selecting action button next to the binary image, selecting download, right click on the download and copy the link. Please note due to the large size of the image the downloading may take a few minutes.
path = 'rock_sample_Leopard' # this is the path folder for reading/saving the image data for this example
name = 'image_Leopard.raw' # this is the name of the saved file
if not os.path.isdir(path):
os.makedirs(path)
file_name = path+'/'+name
url = 'https://www.digitalrocksportal.org/projects/317/images/223481/download/'
def download_image(filename, url):
download_command = f'wget {url} -O {filename}'
try:
subprocess.run(download_command.split(' '))
except FileNotFoundError:
raise InterruptedError(f'wget was not found. Please make sure it is installed on your system.')
return
download_image(file_name, url)
Reading the image as numpy array: After importing the input image, we need to define the dimensions of the sample, this information is usually gathered from the source of the image. Here, a rock sample of size [1000,1000,1000] was used. Next step is to convert the image data to numpy arrays to be compatible with PoreSpy’s input image type for
diffusive_size_factor_AIanddiffusive_size_factor_DNS. Here, numpy.fromfile was used to read the image. More details on this function can be found here
voxelsx = 1000
voxelsy = 1000
voxelsz = 1000
voxel_size = 2.25e-6
im = np.fromfile(file_name, dtype="<i1")
im = np.reshape(im, (voxelsx,voxelsy,voxelsz))
# pore space must be labeled as True and solid phase as False
pore_space = im == 0 # sometimes this may be 255 or some other value depending on the source of the image
im[pore_space] = True
im[~pore_space] = False
print(ps.metrics.porosity(im))
Note: The line (ps.metrics.porosity(im)) is to check the porosity level to the information from the input source description. If there is a significant difference, the labels of the input image may need to be reveresd. e.g. you may need to switch False and True in the code above or replace 0 with a different value. You can check the current values in the loaded image using np.unique(im).
Slicing the image: In this stage we slice the image to a smaller subsection. This is to speed up the process of prediction with the AI approach and the DNS approach.
im = im[:100,:100,:100]
Segmentation of the image#
Snow2 function is part of the PoreSpy’s opensource package, which extracts the pore network and conns from the image.
We then extract the pore network of the porous medium image using PoreSpy’s snow2 algorithm. snow2 returns the segmented image of the porous medium as well as extracted network data.
Size factor prediction#
Create the AI model and load the weights:
model = ps.networks.create_model()
# Giving the path weights to .load function
model.load_weights(path_weights)
2026-11-08 04:21:39.345784: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2026-11-08 04:21:39.407306: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-11-08 04:21:45.411415: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2026-11-08 04:21:46.389673: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
Now that we have the regions, model and image data, we can use it for prediction. Finally we intiate the AI prediction process:
predicted_ai = ps.networks.diffusive_size_factor_AI(
regions, model=model, g_train=g_train, throat_conns=conns, voxel_size=voxel_size
)
1/9 ━━━━━━━━━━━━━━━━━━━━ 28s 4s/step
2/9 ━━━━━━━━━━━━━━━━━━━━ 22s 3s/step
3/9 ━━━━━━━━━━━━━━━━━━━━ 19s 3s/step
4/9 ━━━━━━━━━━━━━━━━━━━━ 15s 3s/step
5/9 ━━━━━━━━━━━━━━━━━━━━ 12s 3s/step
6/9 ━━━━━━━━━━━━━━━━━━━━ 9s 3s/step
7/9 ━━━━━━━━━━━━━━━━━━━━ 6s 3s/step
8/9 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
Cell In[8], line 1
----> 1 predicted_ai = ps.networks.diffusive_size_factor_AI(
2 regions, model=model, g_train=g_train, throat_conns=conns, voxel_size=voxel_size
3 )
File ~/work/porespy/porespy/src/porespy/networks/_size_factors.py:84, in diffusive_size_factor_AI(regions, throat_conns, model, g_train, voxel_size)
82 batch_size = 16
83 test_steps = math.ceil(len(throat_conns) / batch_size)
---> 84 predictions = model.predict(test_data, steps=test_steps)
85 predictions = np.squeeze(predictions)
86 denorm_size_factor = _denorm_predict(predictions, g_train)
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/keras/src/utils/traceback_utils.py:117, in filter_traceback.<locals>.error_handler(*args, **kwargs)
115 filtered_tb = None
116 try:
--> 117 return fn(*args, **kwargs)
118 except Exception as e:
119 filtered_tb = _process_traceback_frames(e.__traceback__)
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/keras/src/backend/tensorflow/trainer.py:588, in TensorFlowTrainer.predict(self, x, batch_size, verbose, steps, callbacks)
586 callbacks.on_predict_batch_begin(begin_step)
587 data = get_data(iterator)
--> 588 batch_outputs = self.predict_function(data)
589 outputs = append_to_outputs(batch_outputs, outputs)
590 callbacks.on_predict_batch_end(
591 end_step, {"outputs": batch_outputs}
592 )
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/util/traceback_utils.py:150, in filter_traceback.<locals>.error_handler(*args, **kwargs)
148 filtered_tb = None
149 try:
--> 150 return fn(*args, **kwargs)
151 except Exception as e:
152 filtered_tb = _process_traceback_frames(e.__traceback__)
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:833, in Function.__call__(self, *args, **kwds)
830 compiler = "xla" if self._jit_compile else "nonXla"
832 with OptionalXlaContext(self._jit_compile):
--> 833 result = self._call(*args, **kwds)
835 new_tracing_count = self.experimental_get_tracing_count()
836 without_tracing = (tracing_count == new_tracing_count)
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:878, in Function._call(self, *args, **kwds)
875 self._lock.release()
876 # In this case we have not created variables on the first call. So we can
877 # run the first trace but we should fail if variables are created.
--> 878 results = tracing_compilation.call_function(
879 args, kwds, self._variable_creation_config
880 )
881 if self._created_variables:
882 raise ValueError("Creating variables on a non-first call to a function"
883 " decorated with tf.function.")
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:139, in call_function(args, kwargs, tracing_options)
137 bound_args = function.function_type.bind(*args, **kwargs)
138 flat_inputs = function.function_type.unpack_inputs(bound_args)
--> 139 return function._call_flat( # pylint: disable=protected-access
140 flat_inputs, captured_inputs=function.captured_inputs
141 )
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/polymorphic_function/concrete_function.py:1322, in ConcreteFunction._call_flat(self, tensor_inputs, captured_inputs)
1318 possible_gradient_type = gradients_util.PossibleTapeGradientTypes(args)
1319 if (possible_gradient_type == gradients_util.POSSIBLE_GRADIENT_TYPES_NONE
1320 and executing_eagerly):
1321 # No tape is watching; skip to running the function.
-> 1322 return self._inference_function.call_preflattened(args)
1323 forward_backward = self._select_forward_and_backward_functions(
1324 args,
1325 possible_gradient_type,
1326 executing_eagerly)
1327 forward_function, args_with_tangents = forward_backward.forward()
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:216, in AtomicFunction.call_preflattened(self, args)
214 def call_preflattened(self, args: Sequence[core.Tensor]) -> Any:
215 """Calls with flattened tensor inputs and returns the structured output."""
--> 216 flat_outputs = self.call_flat(*args)
217 return self.function_type.pack_output(flat_outputs)
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:251, in AtomicFunction.call_flat(self, *args)
249 with record.stop_recording():
250 if self._bound_context.executing_eagerly():
--> 251 outputs = self._bound_context.call_function(
252 self.name,
253 list(args),
254 len(self.function_type.flat_outputs),
255 )
256 else:
257 outputs = make_call_op_in_graph(
258 self,
259 list(args),
260 self._bound_context.function_call_options.as_attrs(),
261 )
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/context.py:1688, in Context.call_function(self, name, tensor_inputs, num_outputs)
1686 cancellation_context = cancellation.context()
1687 if cancellation_context is None:
-> 1688 outputs = execute.execute(
1689 name.decode("utf-8"),
1690 num_outputs=num_outputs,
1691 inputs=tensor_inputs,
1692 attrs=attrs,
1693 ctx=self,
1694 )
1695 else:
1696 outputs = execute.execute_with_cancellation(
1697 name.decode("utf-8"),
1698 num_outputs=num_outputs,
(...) 1702 cancellation_manager=cancellation_context,
1703 )
File ~/work/porespy/porespy/.venv/lib/python3.13/site-packages/tensorflow/python/eager/execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
51 try:
52 ctx.ensure_initialized()
---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
54 inputs, attrs, num_outputs)
55 except core._NotOkStatusException as e:
56 if name is not None:
KeyboardInterrupt:
Similarly we run the DNS(Direct Numerical Simulation) Note: This cell is often the longest to exute. Here for comparison purposes, we used time to calculate the runtime of this cell.
startTime = time.time()
predicted_dns = ps.networks.diffusive_size_factor_DNS(
regions, throat_conns=conns, voxel_size=voxel_size
)
executionTime_dns = time.time() - startTime
print("Execution time in seconds: " + str(executionTime_dns))
Execution time in seconds: 52.41909217834473
Note: Once the values are predicted, you can save them ih hdf5 format for later use.
Finally we plot the Comparison between AI results and DNS predicted results. This helps us to understand the deviation between the two and also mesaure the accuracy and the correctness of the results:
max_val = np.max([predicted_ai, predicted_dns])
plt.figure(figsize=[8, 8])
plt.plot(predicted_ai, predicted_dns, "*", [0, max_val], [0, max_val], "r")
plt.title(r"$R^2$ = " + str(np.round(r2_score(predicted_dns, predicted_ai), 2)))
plt.xlabel("AI_based size factor values")
plt.ylabel("DNS_based size factor values")