WBM filtering fails assert on entries_old_corr

[jackwebersdgr]

I'm attempting to compile the filtered WBM dataset in order to test a new model, but ran into this assert:

matbench-discovery/data/wbm/compile_wbm_test_set.py

Line 531 in c1f34da

assert len(entries_old_corr) == 76_390, f"{len(entries_old_corr)=}, expected 76,390"

assert len(entries_old_corr) == 76_390, f"{len(entries_old_corr)=}, expected 76,390"
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

AssertionError: len(entries_old_corr)=256963, expected 76,390

Is the fully filtered WBM dataset stored anywhere else or must it be computed on the fly using this script? Thanks!

[janosh]

strange, i reran that whole script just last month following #121 and it was back to a working state after #122. either way, no need for you to run that file yourself. the data files it creates are listed here, also up on figshare and will be auto-downloaded if you access the corresponding DataFiles attributes. e.g.

import pandas as pd

from matbench_discovery.data import DataFiles

df_summary = pd.read_csv(DataFiles.wbm_summary.path)
df_wbm_init_structs = pd.read_json(DataFiles.wbm_cses_plus_init_structs.path)

[jackwebersdgr]

Ah, I see. I was under the impression that this script would perform the filtration based on matching structure prototypes in MP, but it seems like this also results in ~257k datapoints. Is there a simple way to obtain the filtered 215.5k set, perhaps via some set of material_ids?

Additionally, it seems the documentation in the site is out of date, and can be updated with the above code https://matbench-discovery.materialsproject.org/contribute#--direct-download

[janosh]

have a look at the subset kwarg in load_df_wbm_with_preds

matbench-discovery/matbench_discovery/preds.py

Lines 90 to 111 in c1f34da

	def load_df_wbm_with_preds(
	*,
	models: Sequence[str] = (),
	pbar: bool = True,
	id_col: str = Key.mat_id,
	subset: pd.Index | Sequence[str] | Literal["uniq_protos"] | None = None,
	max_error_threshold: float | None = 5.0,
	**kwargs: Any,
	) -> pd.DataFrame:
	"""Load WBM summary dataframe with model predictions from disk.
	Args:
	models (Sequence[str], optional): Model names must be keys of
	matbench_discovery.data.Model. Defaults to all models.
	pbar (bool, optional): Whether to show progress bar. Defaults to True.
	id_col (str, optional): Column to set as df.index. Defaults to "material_id".
	subset (pd.Index | Sequence[str] | 'uniq_protos' | None, optional):
	Subset of material IDs to keep. Defaults to None, which loads all materials.
	'uniq_protos' drops WBM structures with matching prototype in MP
	training set and duplicate prototypes in WBM test set (keeping only the most
	stable structure per prototype). This increases the 'OOD-ness' of WBM.
	max_error_threshold (float, optional): Maximum absolute error between predicted

matbench-discovery/matbench_discovery/preds.py

Lines 180 to 182 in c1f34da

	if subset == "uniq_protos":
	df_out = df_out.query(Key.uniq_proto)
	elif subset is not None:

and

matbench-discovery/matbench_discovery/enums.py

Lines 141 to 148 in c1f34da

	@unique
	class TestSubset(LabelEnum):
	"""Which subset of the test data to use for evaluation."""
	uniq_protos = "uniq_protos", "Unique Structure Prototypes"
	ten_k_most_stable = "10k_most_stable", "10k Most Stable"
	full = "full", "Full Test Set"