What’s new in chemfp?

What’s new in chemfp 4.0? (12 June 2022)

The main themes for chemfp 4.0 are “notebook usability” and “diversity selection.”

High-level API

The chemfp API was primarily designed for application developers who want precise control over what happens. The API was used for command-line tools, like the ones that come with chemfp, and for web services.

The command-line tools were written for a wider audience. As a result, I saw that people who used chemfp in a Jupyter notebook, where they had access to the full chemfp API, still preferred “!shell”-ing out to the command-line tools. And rightly so, as it was much simpler.

Chemfp 4.0 includes new “high-level” functionality to remedy this problem. The table below shows which API functions correspond to which command-line tools:

Command-line	High-level API	Description
simsearch	chemfp.simsearch	Similarity search
rdkit2fps	chemfp.rdkit2fps	Use RDKit to generate fingerprints
ob2fps	chemfp.ob2fps	Use Open Babel to generate fingerprints
oe2fps	chemfp.oe2fps	Use OEChem/OEGraphSim to generate fingerprints
cdk2fps	chemfp.cdk2fps	Use CDK to generate fingerprints
(no command-line tool)	chemfp.convert2fps	Figure out which tool to use to generate fingerprints
fpcat	(no high-level API)	Convert between fingerprint file formats
chemfp maxmin	chemfp.maxmin	MaxMin diversity picking
chemfp spherex	chemfp.spherex	Sphere exclusion diversity picking
chemfp heapsweep	chemfp.heapsweep	HeapSweep diversity picking

Here’s an example showing the distribution of fingerprint similarity to caffiene (CHEMBL113) for all fingerprints in ChEMBL 30 which are at least 0.4 similar:

import chemfp

chemfp.simsearch(query_id="CHEMBL113", targets="chembl_30.fpb", threshold=0.4
                ).to_pandas().hist("score", log=True)

array([[<AxesSubplot:title={'center':'score'}>]], dtype=object)

Pandas integration

As you saw in the previous example, chemfp 4.0 add new methods which export chemfp data to a Pandas dataframe. The following shows the nearst 5 neighbors to CHEMBL1113 (using k=6 as caffine is already present):

import chemfp

chemfp.simsearch(query_id="CHEMBL113", targets="chembl_30.fpb", k=6).to_pandas()

	target_id	score
0	CHEMBL113	1.000000
1	CHEMBL4591369	0.733333
2	CHEMBL1232048	0.709677
3	CHEMBL446784	0.677419
4	CHEMBL1738791	0.666667
5	CHEMBL2058173	0.666667

Multi-query search have three columns of output, with the query id in the first:

import chemfp

with chemfp.load_fingerprints("chembl_30.fpb") as arena:
    queries, targets = arena.train_test_split(train_size=10, test_size=100, rng=12345)

result = chemfp.simsearch(queries=queries, targets=targets, k=1, threshold=0.2)
result.to_pandas()

	query_id	target_id	score
0	CHEMBL1316726	CHEMBL2261339	0.224490
1	CHEMBL4588846	CHEMBL495421	0.250000
2	CHEMBL3696763	CHEMBL185354	0.214286
3	CHEMBL4091061	CHEMBL1641956	0.278481
4	CHEMBL2349135	*	NaN
5	CHEMBL1604960	CHEMBL343141	0.315789
6	CHEMBL2012902	*	NaN
7	CHEMBL20406	CHEMBL2322823	0.230769
8	CHEMBL1796344	CHEMBL61106	0.207547
9	CHEMBL3400341	*	NaN

The placeholder target id “*” and score of None (which pandas converts to a NaN) is because three of the compounds had no nearest neighbor with a similarity of at least 0.2.

The column names and placeholders can be changed. The following uses empty = None to not include those queries in the output:

result.to_pandas(columns=["FROM", "TO", "Tanimoto"], empty=None)

	FROM	TO	Tanimoto
0	CHEMBL1316726	CHEMBL2261339	0.224490
1	CHEMBL4588846	CHEMBL495421	0.250000
2	CHEMBL3696763	CHEMBL185354	0.214286
3	CHEMBL4091061	CHEMBL1641956	0.278481
4	CHEMBL1604960	CHEMBL343141	0.315789
5	CHEMBL20406	CHEMBL2322823	0.230769
6	CHEMBL1796344	CHEMBL61106	0.207547

Progress bars

A you saw in the previous example, progress bars have been implemented, both on the command-line and through the high-level API.

Here’s an example using the command-line to generate RDKit fingerprints from a PubChem file:

!rdkit2fps Compound_099000001_099500000.sdf.gz -o Compound_099000001_099500000.fps

Compound_099000001_099500000.sdf.gz: 100%|█| 6.77M/6.77M [00:23<00:00, 282kbytes

and here is the equivalent using the high-level API:

import chemfp
chemfp.rdkit2fps("Compound_099000001_099500000.sdf.gz", "Compound_099000001_099500000.fps")

ConversionInfo("Converted structures from 'Compound_099000001_099500000.sdf.gz'. #input_records=10740, #output_records=10740 (total: 8.19 s)")

The progress bars use tqdm. If tqdm is not installed then chemfp will use a version bundled with chemfp itself.

The default progressbar can be disabled and re-enabled through the command-line:

chemfp.set_default_progressbar(False) # Disable
chemfp.set_default_progressbar(True) # Enable

If the environment variable CHEMFP_PROGRESSBAR is “0” then the default progressbar start in the disabled state.

Improved reprs

Many of the chemfp objects now implement a custom __repr__ which is more useful than the default. For examples:

import chemfp
arena = chemfp.load_fingerprints("chembl_30.fpb")
arena

FPBFingerprintArena(#fingerprints=2136187)

arena.knearest_tanimoto_search_fp(arena.fingerprints[12345], k=5)

SearchResult(#hits=5)

subarena = arena.sample(50, rng=54321)
picks = chemfp.heapsweep(subarena, num_picks=5)
picks

HeapSweepScoreSearch('picked 5 fps. similarity <= 1.0, #candidates=50, seed=-1 (pick: 38.37 ms, total: 38.43 ms)', picker=HeapSweepPicker(#candidates=45, #picks=5), result=PicksAndScores(#picks=5))

result = chemfp.simsearch(queries=subarena, targets=arena, k=3)
result

MultiQuerySimsearch('3-nearest Tanimoto search. #queries=50, #targets=2136187 (search: 848.42 ms total: 848.55 ms)', result=SearchResults(#queries=50, #targets=2136187))

“Shortcut” toolkit imports

Chemfp supports four different cheminformatics toolkits, which it uses for molecule I/O and fingerprint generation. Chemfp also implements a “toolkit” wrapper API, so chemfp-based programs can work with multiple toolkits in a consistent way.

The standard way to access these toolkits is by importing the toolkit wrapper subpackage, like:

from chemfp import openbabel_toolkit
#from chemfp import cdk_toolkit
#from chemfp import openeye_toolkit
#from chemfp import rdkit_toolkit

This adds as a bit of overhead which makes interactive use a bit less enjoyable.

Chemfp 4.0 adds a “shortcut” importer object, where the import occurs the first time it is accessed. For example, here’s a way to get the list of RDKit formats that chemfp supports:

import chemfp
chemfp.rdkit.get_formats()

[Format('rdkit/smi'),
 Format('rdkit/can'),
 Format('rdkit/usm'),
 Format('rdkit/sdf'),
 Format('rdkit/sdf3k'),
 Format('rdkit/smistring'),
 Format('rdkit/canstring'),
 Format('rdkit/usmstring'),
 Format('rdkit/molfile'),
 Format('rdkit/rdbinmol'),
 Format('rdkit/fasta'),
 Format('rdkit/sequence'),
 Format('rdkit/helm'),
 Format('rdkit/mol2'),
 Format('rdkit/pdb'),
 Format('rdkit/xyz'),
 Format('rdkit/mae'),
 Format('rdkit/inchi'),
 Format('rdkit/inchikey'),
 Format('rdkit/inchistring'),
 Format('rdkit/inchikeystring')]

The special importer objects are chemfp.openbabel, chemfp.openeye, chemfp.cdk and chemfp.rdkit.

You should not import these objects as object (like from chemfp import openeye) because you will likely get confused with the real import openeye – I certainly do! Instead, alway use chemfp.openbabel and, if you want to import a specifc toolkit use from chemfp import openbabel_toolkit as ob_toolkit or as T.

NOTE: it seems that Jupyter doesn’t understand how to get the properties of these importer objects as tab completion in the notebook doesn’t work, while it’s just fine in the Python shell.

FingerprintType improvements

A FingerprintType object is the interface to a given toolkit fingerprint type, including its parameters. Before chemfp 4.0 the only way to get a FingerprintType was to use a fingerprint type string, like:

import chemfp
fptype = chemfp.get_fingerprint_type("OpenBabel-FP2")
fptype

OpenBabelFP2FingerprintType_v1(<OpenBabel-FP2/1>)

This was too generic, as it proved difficult to remember the correct string name and it’s parameters.

With chemfp 4.0, each of the toolkit wrapper modules includes a FingerprintType with the default values for each of the toolkit fingerprints families. For the Open Babel example:

from chemfp import openbabel_toolkit
openbabel_toolkit.fp2

OpenBabelFP2FingerprintType_v1(<OpenBabel-FP2/1>)

from chemfp import rdkit_toolkit
rdkit_toolkit.morgan

RDKitMorganFingerprintType_v1(<RDKit-Morgan/1 radius=2 fpSize=2048 useFeatures=0 useChirality=0 useBondTypes=1>)

The newly available FingerprintType objects are:

• chemfp.rdkit_toolkit.avalon - “RDKit-Avalon” fingerprints
• chemfp.rdkit_toolkit.maccs166 - “RDKit-MACCS166” fingerprints
• chemfp.rdkit_toolkit.morgan - “RDKit-Morgan” fingerprints
• chemfp.rdkit_toolkit.atom_pair - “RDKit-AtomPair” fingerprints
• chemfp.rdkit_toolkit.pattern - “RDKit-Pattern” fingerprints
• chemfp.rdkit_toolkit.rdk - “RDKit-Fingerprint” fingerprints
• chemfp.rdkit_toolkit.secfp - “RDKit-SECFP” fingerprints
• chemfp.rdkit_toolkit.torsion - “RDKit-Torsion” fingerprints
• chemfp.openeye_toolkit.circular - “OpenEye-Circular” fingerprints
• chemfp.openeye_toolkit.maccs166 - “OpenEye-MACCS166” fingerprints
• chemfp.openeye_toolkit.mdl_screen - “OpenEye-MDLScreen” fingerprints
• chemfp.openeye_toolkit.molecule_screen- “OpenEye-MoleculeScreen” fingerprints
• chemfp.openeye_toolkit.path - “OpenEye-Path” fingerprints
• chemfp.openeye_toolkit.smarts_screen - “OpenEye-SMARTSScreen” fingerprints
• chemfp.openeye_toolkit.tree -“OpenEye-Tree” fingerprints
• chemfp.openbabel_toolkit.ecfp0 - “OpenBabel-ECFP0” fingerprints
• chemfp.openbabel_toolkit.ecfp2 - “OpenBabel-ECFP2” fingerprints
• chemfp.openbabel_toolkit.ecfp4 - “OpenBabel-ECFP4” fingerprints
• chemfp.openbabel_toolkit.ecfp6 - “OpenBabel-ECFP6” fingerprints
• chemfp.openbabel_toolkit.ecfp8 - “OpenBabel-ECFP8” fingerprints
• chemfp.openbabel_toolkit.ecfp10 - “OpenBabel-ECFP10” fingerprints
• chemfp.openbabel_toolkit.fp2 - “OpenBabel-FP2” fingerprints
• chemfp.openbabel_toolkit.fp3 - “OpenBabel-FP3” fingerprints
• chemfp.openbabel_toolkit.fp4 - “OpenBabel-FP4” fingerprints
• chemfp.openbabel_toolkit.maccs - “OpenBabel-MACCS” fingerprints
• chemfp.cdk_toolkit.atom_pairs2d - “CDK-AtomPairs2D” fingerprints
• chemfp.cdk_toolkit.daylight - “CDK-Daylight” fingerprints
• chemfp.cdk_toolkit.ecfp0 - “CDK-ECFP0” fingerprints
• chemfp.cdk_toolkit.ecfp2 - “CDK-ECFP2” fingerprints
• chemfp.cdk_toolkit.ecfp4 - “CDK-ECFP4” fingerprints
• chemfp.cdk_toolkit.ecfp6 - “CDK-ECFP6” fingerprints
• chemfp.cdk_toolkit.estate - “CDK-EState” fingerprints
• chemfp.cdk_toolkit.extended - “CDK-Extended” fingerprints
• chemfp.cdk_toolkit.fcfp0 - “CDK-FCFP0” fingerprints
• chemfp.cdk_toolkit.fcfp2 - “CDK-FCFP2” fingerprints
• chemfp.cdk_toolkit.fcfp4 - “CDK-FCFP4” fingerprints
• chemfp.cdk_toolkit.fcfp6 - “CDK-FCFP6” fingerprints
• chemfp.cdk_toolkit.graph_only - “CDK-GraphOnly” fingerprints
• chemfp.cdk_toolkit.hybridization - “CDK-Hybridization” fingerprints
• chemfp.cdk_toolkit.klekota_roth - “CDK-KlekotaRoth” fingerprints
• chemfp.cdk_toolkit.maccs - “CDK-MACCS” fingerprints
• chemfp.cdk_toolkit.pubchem - “CDK-Pubchem” fingerprints
• chemfp.cdk_toolkit.shortest_path - “CDK-ShortestPath” fingerprints
• chemfp.cdk_toolkit.substructure - “CDK-Substructure” fingerprints

A FingerprintType is now callable, which lets you make a copy, with some arguments changed.

rdkit_toolkit.morgan(fpSize=128, radius=3)

RDKitMorganFingerprintType_v1(<RDKit-Morgan/1 radius=3 fpSize=128 useFeatures=0 useChirality=0 useBondTypes=1>)

There are new helper methods on the FingerprintType to simplify common cases of working with structure data. For example, the from_smiles method takes a SMILES string as input and generate the corresponding fingerprint.

rdkit_toolkit.morgan(fpSize=128, radius=3).from_smiles("CN1C=NC2=C1C(=O)N(C(=O)N2C)C")

b"\x0b\x06\x01\x08\x93\x10\x19\x04\x00\x84\n@\x10'\x08\x07"

Or, using the shortcut importer and using the InChI string as input:

import chemfp
chemfp.rdkit.morgan(fpSize=128, radius=3).from_inchi(
    "InChI=1S/C8H10N4O2/c1-10-4-9-6-5(10)7(13)12(3)8(14)11(6)2/h4H,1-3H3")

b"\x0b\x06\x01\x08\x93\x10\x19\x04\x00\x84\n@\x10'\x08\x07"

String and file I/O helper functions

There are several helper functions to make it easier to read and write from strings and files.

If you have an FPS file as a string, you can load it with load_fingerprints_from_string:

import chemfp
arena = chemfp.load_fingerprints_from_string("""\
aabbcc\tfirst
10d9b4\tsecond
""")
arena

FingerprintArena(#fingerprints=2)

There are helper in the chemistry toolkit wrapper modules to read and write molecules:

from chemfp import openbabel_toolkit as ob_toolkit
ob_toolkit.from_smiles("c1ccccc1O")

<openbabel.openbabel.OBMol; proxy of <Swig Object of type 'OpenBabel::OBMol *' at 0x117de68b0> >

ob_toolkit.to_inchi(ob_toolkit.from_smiles("c1ccccc1O"))

'InChI=1S/C6H6O/c7-6-4-2-1-3-5-6/h1-5,7H \n'

ob_toolkit.to_sdf3k_file("example.sdf").write_molecule(ob_toolkit.from_smiles("c1ccccc1O"))

==============================
*** Open Babel Warning  in WriteMolecule
  No 2D or 3D coordinates exist. Stereochemical information will be stored using an Open Babel extension. To generate 2D or 3D coordinates instead use --gen2D or --gen3D.

!head example.sdf

 OpenBabel06122218002D

  0  0  0     0  0            999 V3000
M  V30 BEGIN CTAB
M  V30 COUNTS 7 7 0 0 0
M  V30 BEGIN ATOM
M  V30 1 C 0 0 0 0
M  V30 2 C 0 0 0 0
M  V30 3 C 0 0 0 0

“chemfp” command

Chemfp 4.0 added several user-level commands. Rather than create program names which might collide with other name, I decided to add a new “chemfp” command, which implements subcommands. Many of the subcommands are the same as the top-level commands. There are also new commands for diversity search, and a couple related to licensing.

!chemfp

usage: chemfp [--help] [--license-file FILENAME] [--traceback] [--version]
              [command] ...

chemfp tools for cheminformatics fingerprints and structure I/O

positional arguments:
  command
  rest

options:
  --help, -h
  --license-file FILENAME
  --traceback           print the traceback on KeyboardInterrupt
  --version             show program's version number and exit

Summary of the available commands.

Fingerprint generation

  cdk2fps               Generate RDKit fingerprints
  ob2fps                Generate Open Babel fingerprints
  oe2fps                Generate OpenEye fingerprints
  rdkit2fps             Generate RDKit fingerprints
  sdf2fps               Extract a fingerprint tag from an SD file and generate
                         FPS or FPB fingerprints

Fingerprint search

  simsearch             Search an FPS or FPB file for similar fingerprints

Fingerprint format conversion

  fpcat                 Combine multiple fingerprint files into a single file

Fingerprint file tools

  fpb_text              Show the TEXT sections of an FPB file

Diversity selection

  heapsweep             Select diverse fingerprints using the heapsweep
                         algorithm
  maxmin                Select diverse fingerprints using the MaxMin algorithm
  spherex               Select diverse fingerprints using the sphere exclusion
                         algorithm

Configuration

  license               Show the chemfp license status
  report                Report chemfp similarity search implementation details
  toolkits              Show underlying cheminformatics toolkit availability

Diversity selection

Chemfp 4.0 added several forms of diversity picking.

MaxMin (see Ashton et al., https://doi.org/10.1002/qsar.200290002) is an approximate but fast method to select maximally diverse fingerprints from a candidate data set. Chemfp’s version of MaxMin also supports reference-based MaxMin, which selects diverse fingerprints from the candidates which are also diverse from a set of references. (For example, select 1,000 fingerprints from a vendor catalog which most enrich the diversity of a corporate collection.)

The following example uses MaxMin to select 5 diverse compounds from ChEMBL 30 which are also diverse from ChEMBL 29 (this takes over a minute):

!chemfp maxmin -n 5 --references chembl_29.fpb chembl_30.fpb --times

#Diversity/1
#num_bits=2048
#type=maxmin threshold=1.0 num-picks=5 all-equal=0 randomize=1 seed=16034062461884639177
#software=chemfp/4.0
#candidates=chembl_30.fpb
#references=chembl_29.fpb
#date=2022-06-12T16:00:36
i       pick_id score
1       CHEMBL4778247   0.2500000
2       CHEMBL4797319   0.2553191
3       CHEMBL4764617   0.2577320
4       CHEMBL4761572   0.2592593
5       CHEMBL4754099   0.2638889
T_init: 0.02 T_pick: 76.62 #picks: 5 picks/s: 0.07 T_total: 76.63

Heapsweep is an exact but slow method to select maximally diverse fingerprints. It is used to seed the MaxMin algorithm when no initial fingerprint is specified. The 5 globally most diverse fingerprints in ChEMBL 30 are:

!chemfp heapsweep -n 5 chembl_30.fpb --times

#Diversity/1
#num_bits=2048
#type=heapsweep threshold=1.0 num-picks=5 all-equal=0 randomize=1 seed=9508979697301357721
#software=chemfp/4.0
#candidates=chembl_30.fpb
#date=2022-06-12T16:01:54
i       pick_id score
1       CHEMBL4297424   0.0769231
2       CHEMBL2105487   0.0769231
3       CHEMBL1796997   0.0769231
4       CHEMBL1201290   0.0833333
5       CHEMBL4300465   0.0833333
T_init: 0.02 T_pick: 8.75 #picks: 5 picks/s: 0.57 T_total: 8.77

Sphere exclusion (see Hudson et al. https://doi.org/10.1002/qsar.19960150402) is used to reduced some of the clumpiness of random sampling, by avoiding picking fingerprints which are close to previous picks. Chemfp’s version of sphere exclusion also have a reference-based version. Chemfp also implements directed sphere exclusion (DISE) (see Gobbi et al., https://doi.org/10.1021/ci025554v) where a candidate fingerprint where the lowest-rank is chosen, rather than picking from all remaining candidates. The ranks can be assigned by the method of Gobbi et al. or with user-defined ranks.

Here are 10 fingerprints picked from ChEMBL such that no picks are with 0.2 similarity of each other, with the output in csv format, and with a fixed seed:

!chemfp spherex -n 10 --threshold 0.2 --out csv chembl_30.fpb --seed 12345

pick_id,count
CHEMBL1199645,157334
CHEMBL1544208,22982
CHEMBL72725,64693
CHEMBL600500,16792
CHEMBL533667,12671
CHEMBL4073454,28297
CHEMBL3827378,43489
CHEMBL3114290,29501
CHEMBL1184175,25290
CHEMBL566963,45206

CSV and TSV output

The simsearch, maxmin, heapsweep, and spherex commands support alternative output formats, using --out. The two alternatives are “csv” and “tsv”, for comma-separated and tab-separated, and are appropriately quoted for use by Excel. These do not contain a metadata header, and the diversity outputs do not include the pick index. The default output format is “chemfp”.

Here are two examples using simsearch:

!simsearch --query "CN1C=NC2=C1C(=O)N(C(=O)N2C)C" chembl_30.fpb --out csv

!simsearch --query "CN1C=NC2=C1C(=O)N(C(=O)N2C)C" chembl_30.fpb --out tsv

The csv and tsv formats use fixed number of columns, even when the default chemfp format uses a variable number of columns. Instead of having N extra columns for each line, there are N lines, one for each output column.

This can cause problems if N = 0, as when a simsearch query has no matches. In this case there is a synthetic output line with a target id of * and a score of NaN. The simsearch option --empty-target-id and the spherex option --empty-hit-id change the default id. The option --empty-score changes the default score.

Use --no-include-empty to not include a synthetic line when N = 0.

Miscellaneous

• Use the output format “sdf3k” to always write SD files in V3000 format.
• The low-level k-nearest arena query search API accepts an initial array of minimum thresholds, one per query. This may be useful when searching multiple target arenas, as the scores from earlier results may help reduce the search space.
• Fingerprint arenas have a new get_bit_counts() method which for each bit counts the number of fingerprints where that bit is on.
• Added open_from_string() and load_fingerprints_from_string() to make it easier to work with FPS or FPB content as a string.
• Added support for Python 3.10 and CDK 2.7.
• The SearchResult has a query_id, if known.
• The NxN and NxM arena similarity search functions in chemfp.search now support optional batch_size and batch_callback parameters. These are used to implement progress bars.
• The Location object now also supports position (the approximate location in the input file), end_position (the expected end position, or None), and position_units (currently only “bytes”). These are used for the progress bar.
• The FPB writers have a location, which the high-level conversion functions used to get the number of output records.
• Added byte_xor_popcount, hex_xor_popcount, byte_union_popcount, and hex_union_popcount to the bitops module.
• Changed CDK-ShortestPath version to “2.7” because the PRNG change in 2.7 results in new bit patterns.
• Added CDK ExtendedFingerprinter support for CDK 2.5 and later.

I also split the C extension into several extensions, including some based on Cython, and I dropped most of the slowest popcount implementations on the assumption that __builtin_popcountll is always good enough.

Breaking changes

Chemfp 4.0 drops support for Python 2.7, for Python 3.7 or earlier, and for toolkits versions before 2020.

Nearly everything else is backwards compatible with older versions of chemfp. There are three changes which I hope breaks no existing code, and if it does, they should be easy to fix.

1. At the API level, the default similarity search had k=3 nearest neighbors with threshold=0.7 as Python default parameters. This kept tripping me up when I because when I wanted (say) all k=10 nearest neigbhors I would forget to change the threshold to 0.0. The default now is that if neither k nor threshold are specified then do a k=3 and threshold=0.7 search. If k is specified and threshold is not, use threshold=0.0. This matches the simsearch command-line behavior.
2. The SearchResults.reorder_all() order parameter has been renamed ‘ordering’ be consistent with SearchResult.reorder() and with internal use.
3. The “difference” functions in chemfp.bitops function have been renamed to xor because even I couldn’t remember what “difference” meant. The xor fuction names are byte_xor and hex_xor.

Deprecation warning

The FingeprintType method compute_fingerprint and compute_fingerprints are deprecated in favor of the have been deprecated in favor of the names from_mol and from_mols. The old methods will be removed in a a future version of chemfp.

Older news (pre 4.0)

To learn about what was new in older chemfp releases, see the What’s New documentation from chemfp 3.5.