.. _whats_new_40:

What’s new in chemfp 4.0
========================

Released 12 June 2022.

The main themes for chemfp 4.0 are “notebook usability” and “diversity
selection.”

(Note: this has been edited on 24 April 2023 to reflect API changes in
chemfp 4.1.)

High-level API
--------------

The chemfp API was primarily designed for application developers who
want precise control over what happens. The API was used for
command-line tools, like `the ones that come with chemfp <tools.rst>`__,
and for web services.

The command-line tools were written for a wider audience. As a result, I
saw that people who used chemfp in a Jupyter notebook, where they had
access to the full chemfp API, still preferred “!shell”-ing out to the
command-line tools. And rightly so, as it was much simpler.

Chemfp 4.0 includes new “high-level” functionality to remedy this
problem. The table below shows which API functions correspond to which
command-line tools:

+------------------------------------------+----------------------------+-------------------------------------------------------+
| Command-line                             | High-level API             | Description                                           |
+==========================================+============================+=======================================================+
| :ref:`simsearch <simsearch>`             | :func:`chemfp.simsearch`   | Similarity search                                     |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`rdkit2fps <rdkit2fps>`             | :func:`chemfp.rdkit2fps`   | Use RDKit to generate fingerprints                    |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`ob2fps <ob2fps>`                   | :func:`chemfp.ob2fps`      | Use Open Babel to generate fingerprints               |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`oe2fps <oe2fps>`                   | :func:`chemfp.oe2fps`      | Use OEChem/OEGraphSim to generate fingerprints        |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`cdk2fps <cdk2fps>`                 | :func:`chemfp.cdk2fps`     | Use CDK to generate fingerprints                      |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| (no command-line tool)                   | :func:`chemfp.convert2fps` | Figure out which tool to use to generate fingerprints |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`fpcat <fpcat>`                     | (no high-level API)        | Convert between fingerprint file formats              |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`chemfp maxmin <chemfp_maxmin>`     | :func:`chemfp.maxmin`      | MaxMin diversity picking                              |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`chemfp spherex <chemfp_spherex>`   | :func:`chemfp.spherex`     | Sphere exclusion diversity picking                    |
+------------------------------------------+----------------------------+-------------------------------------------------------+
| :ref:`chemfp heapsweep <chemfp_spherex>` | :func:`chemfp.heapsweep`   | HeapSweep diversity picking                           |
+------------------------------------------+----------------------------+-------------------------------------------------------+


Here’s an example showing the distribution of fingerprint similarity to
caffiene (CHEMBL113) for all fingerprints in ChEMBL 30 which are at
least 0.4 similar:

.. code-block:: python

    import chemfp
    
    chemfp.simsearch(query_id="CHEMBL113", targets="chembl_30.fpb", threshold=0.4
                    ).to_pandas().hist("score", log=True)




.. parsed-literal::

    array([[<Axes: title={'center': 'score'}>]], dtype=object)




.. image:: whatsnew_files/whatsnew_3_1.png


Pandas integration
------------------

As you saw in the previous example, chemfp 4.0 add new methods which
export chemfp data to a Pandas dataframe. The following shows the nearst
5 neighbors to CHEMBL1113 (using k=6 as caffine is already present):

.. code-block:: python

    import chemfp
    
    chemfp.simsearch(query_id="CHEMBL113", targets="chembl_30.fpb", k=6).to_pandas()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>target_id</th>
          <th>score</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>CHEMBL113</td>
          <td>1.000000</td>
        </tr>
        <tr>
          <th>1</th>
          <td>CHEMBL4591369</td>
          <td>0.733333</td>
        </tr>
        <tr>
          <th>2</th>
          <td>CHEMBL1232048</td>
          <td>0.709677</td>
        </tr>
        <tr>
          <th>3</th>
          <td>CHEMBL446784</td>
          <td>0.677419</td>
        </tr>
        <tr>
          <th>4</th>
          <td>CHEMBL1738791</td>
          <td>0.666667</td>
        </tr>
        <tr>
          <th>5</th>
          <td>CHEMBL2058173</td>
          <td>0.666667</td>
        </tr>
      </tbody>
    </table>
    </div>



Multi-query search have three columns of output, with the query id in
the first:

.. code-block:: python

    import chemfp
    
    with chemfp.load_fingerprints("chembl_30.fpb") as arena:
        queries, targets = arena.train_test_split(train_size=10, test_size=100, rng=12345)
    
    result = chemfp.simsearch(queries=queries, targets=targets, k=1, threshold=0.2)
    result.to_pandas()



.. parsed-literal::

    queries:   0%|                                                                         | 0/10 [00:00<?, ? fps/…




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>query_id</th>
          <th>target_id</th>
          <th>score</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>CHEMBL1316726</td>
          <td>CHEMBL2261339</td>
          <td>0.224490</td>
        </tr>
        <tr>
          <th>1</th>
          <td>CHEMBL4588846</td>
          <td>CHEMBL495421</td>
          <td>0.250000</td>
        </tr>
        <tr>
          <th>2</th>
          <td>CHEMBL3696763</td>
          <td>CHEMBL185354</td>
          <td>0.214286</td>
        </tr>
        <tr>
          <th>3</th>
          <td>CHEMBL4091061</td>
          <td>CHEMBL1641956</td>
          <td>0.278481</td>
        </tr>
        <tr>
          <th>4</th>
          <td>CHEMBL2349135</td>
          <td>*</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>5</th>
          <td>CHEMBL1604960</td>
          <td>CHEMBL343141</td>
          <td>0.315789</td>
        </tr>
        <tr>
          <th>6</th>
          <td>CHEMBL2012902</td>
          <td>*</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>7</th>
          <td>CHEMBL20406</td>
          <td>CHEMBL2322823</td>
          <td>0.230769</td>
        </tr>
        <tr>
          <th>8</th>
          <td>CHEMBL1796344</td>
          <td>CHEMBL61106</td>
          <td>0.207547</td>
        </tr>
        <tr>
          <th>9</th>
          <td>CHEMBL3400341</td>
          <td>*</td>
          <td>NaN</td>
        </tr>
      </tbody>
    </table>
    </div>



The placeholder target id “\*” and score of None (which pandas converts
to a NaN) is because three of the compounds had no nearest neighbor with
a similarity of at least 0.2.

The column names and placeholders can be changed. The following uses
``empty = None`` to not include those queries in the output:

.. code-block:: python

    result.to_pandas(columns=["FROM", "TO", "Tanimoto"], empty=None)




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>FROM</th>
          <th>TO</th>
          <th>Tanimoto</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>CHEMBL1316726</td>
          <td>CHEMBL2261339</td>
          <td>0.224490</td>
        </tr>
        <tr>
          <th>1</th>
          <td>CHEMBL4588846</td>
          <td>CHEMBL495421</td>
          <td>0.250000</td>
        </tr>
        <tr>
          <th>2</th>
          <td>CHEMBL3696763</td>
          <td>CHEMBL185354</td>
          <td>0.214286</td>
        </tr>
        <tr>
          <th>3</th>
          <td>CHEMBL4091061</td>
          <td>CHEMBL1641956</td>
          <td>0.278481</td>
        </tr>
        <tr>
          <th>4</th>
          <td>CHEMBL1604960</td>
          <td>CHEMBL343141</td>
          <td>0.315789</td>
        </tr>
        <tr>
          <th>5</th>
          <td>CHEMBL20406</td>
          <td>CHEMBL2322823</td>
          <td>0.230769</td>
        </tr>
        <tr>
          <th>6</th>
          <td>CHEMBL1796344</td>
          <td>CHEMBL61106</td>
          <td>0.207547</td>
        </tr>
      </tbody>
    </table>
    </div>



Progress bars
-------------

A you saw in the previous example, progress bars have been implemented,
both on the command-line and through the high-level API.

Here’s an example using the command-line to generate RDKit fingerprints
from a PubChem file:

.. code-block:: console

    !rdkit2fps Compound_099000001_099500000.sdf.gz -o Compound_099000001_099500000.fps


.. parsed-literal::

    Compound_099000001_099500000.sdf.gz: 100%|█| 6.77M/6.77M [00:10<00:00, 626kbytes


and here is the equivalent using the high-level API:

.. code-block:: python3

    import chemfp
    chemfp.rdkit2fps("Compound_099000001_099500000.sdf.gz", "Compound_099000001_099500000.fps")



.. parsed-literal::

    Compound_099000001_099500000.sdf.gz:   0%|                                      | 0.00/6.77M [00:00<?, ?bytes/…




.. parsed-literal::

    ConversionInfo("Converted structures from 'Compound_099000001_099500000.sdf.gz'. #input_records=10740, #output_records=10740 (total: 3.67 s)")



The progress bars use `tqdm <https://tqdm.github.io/>`__, which (EDIT
for chemfp 4.1) is a required chemfp install dependency.

The default progressbar can be disabled and re-enabled through the
command-line:

.. code-block:: python

    chemfp.set_default_progressbar(False) # Disable
    chemfp.set_default_progressbar(True) # Enable

If the environment variable CHEMFP_PROGRESSBAR is “0” then the default
progressbar start in the disabled state.

Improved ``repr``\ s
--------------------

Many of the chemfp objects now implement a custom ``__repr__`` which is
more useful than the default. For examples:

.. code-block:: python

    import chemfp
    arena = chemfp.load_fingerprints("chembl_30.fpb")
    arena




.. parsed-literal::

    FPBFingerprintArena(#fingerprints=2136187)



.. code-block:: python

    arena.knearest_tanimoto_search_fp(arena.fingerprints[12345], k=5)




.. parsed-literal::

    SearchResult(#hits=5)



.. code-block:: python

    subarena = arena.sample(50, rng=54321)
    picks = chemfp.heapsweep(subarena, num_picks=5)
    picks



.. parsed-literal::

    picks:   0%|                                                                                | 0/5 [00:00<?, ?/…




.. parsed-literal::

    HeapSweepScoreSearch('picked 5 fps. similarity <= 1.0, #candidates=50, seed=-1 (pick: 7.87 ms, total: 8.17 ms)', picker=HeapSweepPicker(#candidates=45, #picks=5), result=PicksAndScores(#picks=5))



.. code-block:: python

    result = chemfp.simsearch(queries=subarena, targets=arena, k=3)
    result



.. parsed-literal::

    queries:   0%|                                                                         | 0/50 [00:00<?, ? fps/…




.. parsed-literal::

    MultiQuerySimsearch('3-nearest Tanimoto search. #queries=50, #targets=2136187 (search: 373.85 ms total: 373.88 ms)', result=SearchResults(#queries=50, #targets=2136187))



“Shortcut” toolkit imports
--------------------------

Chemfp supports four different cheminformatics toolkits, which it uses
for molecule I/O and fingerprint generation. Chemfp also implements a
“toolkit” wrapper API, so chemfp-based programs can work with multiple
toolkits in a consistent way.

The standard way to access these toolkits is by importing the toolkit
wrapper subpackage, like:

.. code-block:: python

    from chemfp import openbabel_toolkit
    #from chemfp import cdk_toolkit
    #from chemfp import openeye_toolkit
    #from chemfp import rdkit_toolkit

This adds as a bit of overhead which makes interactive use a bit less
enjoyable.

Chemfp 4.0 adds a “shortcut” importer object, where the import occurs
the first time it is accessed. For example, here’s a way to get the list
of RDKit formats that chemfp supports:

.. code-block:: python

    import chemfp
    chemfp.rdkit.get_formats()




.. parsed-literal::

    [Format('rdkit/smi'),
     Format('rdkit/can'),
     Format('rdkit/usm'),
     Format('rdkit/cxsmi'),
     Format('rdkit/sdf'),
     Format('rdkit/sdf3k'),
     Format('rdkit/smistring'),
     Format('rdkit/canstring'),
     Format('rdkit/usmstring'),
     Format('rdkit/cxsmistring'),
     Format('rdkit/molfile'),
     Format('rdkit/rdbinmol'),
     Format('rdkit/fasta'),
     Format('rdkit/sequence'),
     Format('rdkit/helm'),
     Format('rdkit/mol2'),
     Format('rdkit/pdb'),
     Format('rdkit/xyz'),
     Format('rdkit/mae'),
     Format('rdkit/inchi'),
     Format('rdkit/inchikey'),
     Format('rdkit/inchistring'),
     Format('rdkit/inchikeystring')]



The special importer objects are :attr:`chemfp.openbabel`,
:attr:`chemfp.openeye`, :attr:`chemfp.cdk` and :attr:`chemfp.rdkit`.

You should not import these objects as object (like
``from chemfp import openeye``) because you will likely get confused
with the real ``import openeye`` – I certainly do! Instead, alway use
``chemfp.openbabel`` and, if you want to import a specific toolkit use
``from chemfp import openbabel_toolkit as ob_toolkit`` or ``as T``.

NOTE: it seems that Jupyter doesn’t understand how to get the properties
of these importer objects as tab completion in the notebook doesn’t
work, while it’s just fine in the Python shell.

FingerprintType improvements
----------------------------

A ``FingerprintType`` object is the interface to a given toolkit
fingerprint type, including its parameters. Before chemfp 4.0 the only
way to get a FingerprintType was to use a fingerprint type string, like:

.. code-block:: python

    import chemfp
    fptype = chemfp.get_fingerprint_type("OpenBabel-FP2")
    fptype




.. parsed-literal::

    OpenBabelFP2FingerprintType_v1(<OpenBabel-FP2/1>)



This was too generic, as it proved difficult to remember the correct
string name and it’s parameters.

With chemfp 4.0, each of the toolkit wrapper modules includes a
FingerprintType with the default values for each of the toolkit
fingerprints families. For the Open Babel example:

.. code-block:: python

    from chemfp import openbabel_toolkit
    openbabel_toolkit.fp2




.. parsed-literal::

    OpenBabelFP2FingerprintType_v1(<OpenBabel-FP2/1>)



.. code-block:: python

    from chemfp import rdkit_toolkit
    rdkit_toolkit.morgan




.. parsed-literal::

    RDKitMorganFingerprintType_v1(<RDKit-Morgan/1 radius=2 fpSize=2048 useFeatures=0 useChirality=0 useBondTypes=1>)



The newly available FingerprintType objects are:

-  :attr:`chemfp.rdkit_toolkit.avalon` - “RDKit-Avalon” fingerprints
-  :attr:`chemfp.rdkit_toolkit.maccs166` - “RDKit-MACCS166” fingerprints
-  :attr:`chemfp.rdkit_toolkit.morgan` - “RDKit-Morgan” fingerprints,
   radius 2, alias for ``morgan2``
-  :attr:`chemfp.rdkit_toolkit.morgan1` - “RDKit-Morgan” fingerprints,
   radius 1 (added in chemfp 4.1) 
-  :attr:`chemfp.rdkit_toolkit.morgan2`- “RDKit-Morgan” fingerprints
   radius 2 (added in chemfp 4.1) 
-  :attr:`chemfp.rdkit_toolkit.morgan3` - “RDKit-Morgan” fingerprints
   radius 3 (added in chemfp 4.1) 
-  :attr:`chemfp.rdkit_toolkit.morgan4` - “RDKit-Morgan” fingerprints
   radius 4 (added in chemfp 4.1) 
-  :attr:`chemfp.rdkit_toolkit.atom_pair` - “RDKit-AtomPair” fingerprints
-  :attr:`chemfp.rdkit_toolkit.pattern` - “RDKit-Pattern” fingerprints
-  :attr:`chemfp.rdkit_toolkit.rdk` - “RDKit-Fingerprint” fingerprints
-  :attr:`chemfp.rdkit_toolkit.secfp` - “RDKit-SECFP” fingerprints
-  :attr:`chemfp.rdkit_toolkit.torsion` - “RDKit-Torsion” fingerprints
-  :attr:`chemfp.openeye_toolkit.circular` - “OpenEye-Circular” fingerprints
-  :attr:`chemfp.openeye_toolkit.maccs166` - “OpenEye-MACCS166” fingerprints
-  :attr:`chemfp.openeye_toolkit.mdl_screen` - “OpenEye-MDLScreen” fingerprints
-  :attr:`chemfp.openeye_toolkit.molecule_screen` - “OpenEye-MoleculeScreen” fingerprints
-  :attr:`chemfp.openeye_toolkit.path` - “OpenEye-Path” fingerprints
-  :attr:`chemfp.openeye_toolkit.smarts_screen` - “OpenEye-SMARTSScreen” fingerprints
-  :attr:`chemfp.openeye_toolkit.tree` - “OpenEye-Tree” fingerprints
-  :attr:`chemfp.openbabel_toolkit.ecfp0` - “OpenBabel-ECFP0” fingerprints
-  :attr:`chemfp.openbabel_toolkit.ecfp2` - “OpenBabel-ECFP2” fingerprints
-  :attr:`chemfp.openbabel_toolkit.ecfp4` - “OpenBabel-ECFP4” fingerprints
-  :attr:`chemfp.openbabel_toolkit.ecfp6` - “OpenBabel-ECFP6” fingerprints
-  :attr:`chemfp.openbabel_toolkit.ecfp8` - “OpenBabel-ECFP8” fingerprints
-  :attr:`chemfp.openbabel_toolkit.ecfp10` - “OpenBabel-ECFP10” fingerprints
-  :attr:`chemfp.openbabel_toolkit.fp2` - “OpenBabel-FP2” fingerprints
-  :attr:`chemfp.openbabel_toolkit.fp3` - “OpenBabel-FP3” fingerprints
-  :attr:`chemfp.openbabel_toolkit.fp4` - “OpenBabel-FP4” fingerprints
-  :attr:`chemfp.openbabel_toolkit.maccs` - “OpenBabel-MACCS” fingerprints
-  :attr:`chemfp.cdk_toolkit.atom_pairs2d` - “CDK-AtomPairs2D” fingerprints
-  :attr:`chemfp.cdk_toolkit.daylight` - “CDK-Daylight” fingerprints
-  :attr:`chemfp.cdk_toolkit.ecfp0` - “CDK-ECFP0” fingerprints
-  :attr:`chemfp.cdk_toolkit.ecfp2` - “CDK-ECFP2” fingerprints
-  :attr:`chemfp.cdk_toolkit.ecfp4` - “CDK-ECFP4” fingerprints
-  :attr:`chemfp.cdk_toolkit.ecfp6` - “CDK-ECFP6” fingerprints
-  :attr:`chemfp.cdk_toolkit.estate` - “CDK-EState” fingerprints
-  :attr:`chemfp.cdk_toolkit.extended` - “CDK-Extended” fingerprints
-  :attr:`chemfp.cdk_toolkit.fcfp0` - “CDK-FCFP0” fingerprints
-  :attr:`chemfp.cdk_toolkit.fcfp2` - “CDK-FCFP2” fingerprints
-  :attr:`chemfp.cdk_toolkit.fcfp4` - “CDK-FCFP4” fingerprints
-  :attr:`chemfp.cdk_toolkit.fcfp6` - “CDK-FCFP6” fingerprints
-  :attr:`chemfp.cdk_toolkit.graph_only` - “CDK-GraphOnly” fingerprints
-  :attr:`chemfp.cdk_toolkit.hybridization` - “CDK-Hybridization” fingerprints
-  :attr:`chemfp.cdk_toolkit.klekota_roth` - “CDK-KlekotaRoth” fingerprints
-  :attr:`chemfp.cdk_toolkit.maccs` - “CDK-MACCS” fingerprints
-  :attr:`chemfp.cdk_toolkit.pubchem` - “CDK-Pubchem” fingerprints
-  :attr:`chemfp.cdk_toolkit.shortest_path` - “CDK-ShortestPath” fingerprints
-  :attr:`chemfp.cdk_toolkit.substructure` - “CDK-Substructure” fingerprints

A FingerprintType is now callable, which lets you make a copy, with some
arguments changed.

.. code-block:: python

    rdkit_toolkit.morgan(fpSize=128, radius=3)




.. parsed-literal::

    RDKitMorganFingerprintType_v1(<RDKit-Morgan/1 radius=3 fpSize=128 useFeatures=0 useChirality=0 useBondTypes=1>)



There are new helper methods on the FingerprintType to simplify common
cases of working with structure data. For example, the ``from_smiles``
method takes a SMILES string as input and generate the corresponding
fingerprint.

.. code-block:: python

    rdkit_toolkit.morgan(fpSize=128, radius=3).from_smiles("CN1C=NC2=C1C(=O)N(C(=O)N2C)C")




.. parsed-literal::

    b"\x0b\x06\x01\x08\x93\x10\x19\x04\x00\x84\n@\x10'\x08\x07"



Or, using the shortcut importer and using the InChI string as input:

.. code-block:: python

    import chemfp
    chemfp.rdkit.morgan(fpSize=128, radius=3).from_inchi(
        "InChI=1S/C8H10N4O2/c1-10-4-9-6-5(10)7(13)12(3)8(14)11(6)2/h4H,1-3H3")




.. parsed-literal::

    b"\x0b\x06\x01\x08\x93\x10\x19\x04\x00\x84\n@\x10'\x08\x07"



String and file I/O helper functions
------------------------------------

There are several helper functions to make it easier to read and write
from strings and files.

If you have an FPS file as a string, you can load it with
``load_fingerprints_from_string``:

.. code-block:: python

    import chemfp
    arena = chemfp.load_fingerprints_from_string("""\
    aabbcc\tfirst
    10d9b4\tsecond
    """)
    arena




.. parsed-literal::

    FingerprintArena(#fingerprints=2)



There are helpers in the chemistry toolkit wrapper modules to read and
write molecules:

.. code-block:: python

    from chemfp import openbabel_toolkit as ob_toolkit
    ob_toolkit.parse_smiles("c1ccccc1O")




.. parsed-literal::

    <openbabel.openbabel.OBMol; proxy of <Swig Object of type 'OpenBabel::OBMol *' at 0x115d2c180> >



.. code-block:: python

    ob_toolkit.create_inchi(ob_toolkit.parse_smiles("c1ccccc1O"))




.. parsed-literal::

    'InChI=1S/C6H6O/c7-6-4-2-1-3-5-6/h1-5,7H \n'



.. code-block:: python

    ob_toolkit.open_sdf3k_writer("example.sdf").write_molecule(ob_toolkit.parse_smiles("c1ccccc1O"))


.. parsed-literal::

    ==============================
    *** Open Babel Warning  in WriteMolecule
      No 2D or 3D coordinates exist. Stereochemical information will be stored using an Open Babel extension. To generate 2D or 3D coordinates instead use --gen2D or --gen3D.


.. code-block:: console

    !head example.sdf


.. parsed-literal::

    
     OpenBabel04242317452D
    
      0  0  0     0  0            999 V3000
    M  V30 BEGIN CTAB
    M  V30 COUNTS 7 7 0 0 0
    M  V30 BEGIN ATOM
    M  V30 1 C 0 0 0 0
    M  V30 2 C 0 0 0 0
    M  V30 3 C 0 0 0 0


“chemfp” command
----------------

Chemfp 4.0 added several user-level commands. Rather than create program
names which might collide with other name, I decided to add a new
“chemfp” command, which implements subcommands. Many of the subcommands
are the same as the top-level commands. There are also new commands for
diversity search, and a couple related to licensing.

.. code-block:: console

    !chemfp


.. parsed-literal::

    Usage: chemfp [OPTIONS] COMMAND [ARGS]...
    
    Options:
      --version                Show the version and exit.
      --traceback              Print the traceback on KeyboardInterrupt
      --license-file FILENAME  Specify a chemfp license file
      --help                   Show this message and exit.
    
    Generation commands:
      cdk2fps    Generate fingerprints using CDK.
      ob2fps     Generate fingerprints using Open Babel.
      oe2fps     Generate fingerprints using OEChem and OEGraphSim.
      rdkit2fps  Generate fingerprints using RDKit.
      sdf2fps    Extract fingerprints from an SDF file.
    
    Algorithms:
      simsearch  Search an FPS or FPB file for similar fingerprints.
      spherex    Diversity selection using the sphere exclusion algorithm.
      maxmin     Diversity selection using the MaxMin algorithm.
      heapsweep  Diversity selection using the heapsweep algorithm.
    
    Fingerprint file commands:
      fpcat     Combine multiple fingerprint files into a single file.
      fpb_text  Show the TEXT sections of an FPB file.
    
    Other commands:
      license   Show the chemfp license status.
      report    Report chemfp similarity search implementation details.
      toolkits  Show underlying cheminformatics toolkit availability.


Diversity selection
~~~~~~~~~~~~~~~~~~~

Chemfp 4.0 added several forms of diversity picking.

MaxMin (see Ashton et al., https://doi.org/10.1002/qsar.200290002) is an
approximate but fast method to select maximally diverse fingerprints
from a candidate data set. Chemfp’s version of MaxMin also supports
reference-based MaxMin, which selects diverse fingerprints from the
candidates which are also diverse from a set of references. (For
example, select 1,000 fingerprints from a vendor catalog which most
enrich the diversity of a corporate collection.)

The following example uses MaxMin to select 5 diverse compounds from
ChEMBL 30 which are also diverse from ChEMBL 29 (this takes over a
minute):

.. code-block:: console

    !chemfp maxmin -n 5 --references chembl_29.fpb chembl_30.fpb --times


.. parsed-literal::

    #Diversity/1
    #num_bits=2048
    #type=maxmin threshold=1.0 num-picks=5 all-equal=0 randomize=1 seed=1397588774
    #software=chemfp/4.0
    #candidates=chembl_30.fpb
    #references=chembl_29.fpb
    #date=2023-04-24T15:45:11
    i	pick_id	score
    1	CHEMBL4778247	0.2500000
    2	CHEMBL4797319	0.2553191
    3	CHEMBL4764617	0.2577320
    4	CHEMBL4761572	0.2592593
    5	CHEMBL4754099	0.2638889
    T_init: 0.02 T_pick: 75.65 #picks: 5 picks/s: 0.07 T_total: 75.68


Heapsweep is an exact but slow method to select maximally diverse
fingerprints. It is used to seed the MaxMin algorithm when no initial
fingerprint is specified. The 5 globally most diverse fingerprints in
ChEMBL 30 are:

.. code-block:: console

    !chemfp heapsweep -n 5 chembl_30.fpb --times


.. parsed-literal::

    #Diversity/1
    #num_bits=2048
    #type=heapsweep threshold=1.0 num-picks=5 all-equal=0 randomize=1 seed=2006307975
    #software=chemfp/4.0
    #candidates=chembl_30.fpb
    #date=2023-04-24T15:46:27
    i	pick_id	score
    1	CHEMBL1796997	0.0769231
    2	CHEMBL4297424	0.0769231
    3	CHEMBL2105487	0.0769231
    4	CHEMBL1201290	0.0833333
    5	CHEMBL4300465	0.0833333
    T_init: 0.03 T_pick: 6.92 #picks: 5 picks/s: 0.72 T_total: 6.97


Sphere exclusion (see Hudson et
al. https://doi.org/10.1002/qsar.19960150402) is used to reduced some of
the clumpiness of random sampling, by avoiding picking fingerprints
which are close to previous picks. Chemfp’s version of sphere exclusion
also have a reference-based version. Chemfp also implements directed
sphere exclusion (DISE) (see Gobbi et al.,
https://doi.org/10.1021/ci025554v) where a candidate fingerprint where
the lowest-rank is chosen, rather than picking from all remaining
candidates. The ranks can be assigned by the method of Gobbi et al. or
with user-defined ranks.

Here are 10 fingerprints picked from ChEMBL such that no picks are with
0.2 similarity of each other, with the output in csv format, and with a
fixed seed:

.. code-block:: console

    !chemfp spherex -n 10 --threshold 0.2 --out csv chembl_30.fpb --seed 12345


.. parsed-literal::

    pick_id,count
    CHEMBL1199645,157334
    CHEMBL1544208,22982
    CHEMBL72725,64693
    CHEMBL600500,16792
    CHEMBL533667,12671
    CHEMBL4073454,28297
    CHEMBL3827378,43489
    CHEMBL3114290,29501
    CHEMBL1184175,25290
    CHEMBL566963,45206


CSV and TSV output
------------------

The simsearch, maxmin, heapsweep, and spherex commands support
alternative output formats, using ``--out``. The two alternatives are
“csv” and “tsv”, for comma-separated and tab-separated, and are
appropriately quoted for use by Excel. These do not contain a metadata
header, and the diversity outputs do not include the pick index. The
default output format is “chemfp”.

Here are two examples using simsearch:

.. code-block:: console

    !simsearch --query "CN1C=NC2=C1C(=O)N(C(=O)N2C)C" chembl_30.fpb --out csv


.. parsed-literal::

    query_id,target_id,score
    Query1,CHEMBL113,1.0000000
    Query1,CHEMBL4591369,0.7333333
    Query1,CHEMBL1232048,0.7096774


.. code-block:: console

    !simsearch --query "CN1C=NC2=C1C(=O)N(C(=O)N2C)C" chembl_30.fpb --out tsv


.. parsed-literal::

    query_id	target_id	score
    Query1	CHEMBL113	1.0000000
    Query1	CHEMBL4591369	0.7333333
    Query1	CHEMBL1232048	0.7096774


The csv and tsv formats use fixed number of columns, even when the
default chemfp format uses a variable number of columns. Instead of
having N extra columns for each line, there are N lines, one for each
output column.

This can cause problems if N = 0, as when a simsearch query has no
matches. In this case there is a synthetic output line with a target id
of ``*`` and a score of ``NaN``. The simsearch option
``--empty-target-id`` and the spherex option ``--empty-hit-id`` change
the default id. The option ``--empty-score`` changes the default score.

Use ``--no-include-empty`` to not include a synthetic line when N = 0.

Miscellaneous
-------------

-  Use the output format “sdf3k” to always write SD files in V3000
   format.

-  The low-level k-nearest arena query search API accepts an initial
   array of minimum thresholds, one per query. This may be useful when
   searching multiple target arenas, as the scores from earlier results
   may help reduce the search space.

-  Fingerprint arenas have a new ``get_bit_counts()`` method which for
   each bit counts the number of fingerprints where that bit is on.

-  Added ``open_from_string()`` and ``load_fingerprints_from_string()``
   to make it easier to work with FPS or FPB content as a string.

-  Added support for Python 3.10 and CDK 2.7.

-  The SearchResult has a ``query_id``, if known.

-  The NxN and NxM arena similarity search functions in
   ``chemfp.search`` now support optional ``batch_size`` and
   ``batch_callback`` parameters. These are used to implement progress
   bars.

-  The Location object now also supports ``position`` (the approximate
   location in the input file), ``end_position`` (the expected end
   position, or None), and ``position_units`` (currently only “bytes”).
   These are used for the progress bar.

-  The FPB writers have a ``location``, which the high-level conversion
   functions used to get the number of output records.

-  Added ``byte_xor_popcount``, ``hex_xor_popcount``,
   ``byte_union_popcount``, and ``hex_union_popcount`` to the bitops
   module.

-  Changed CDK-ShortestPath version to “2.7” because the PRNG change in
   2.7 results in new bit patterns.

-  Added CDK ExtendedFingerprinter support for CDK 2.5 and later.

I also split the C extension into several extensions, including some
based on Cython, and I dropped most of the slowest popcount
implementations on the assumption that ``__builtin_popcountll`` is
always good enough.

Breaking changes
----------------

Chemfp 4.0 drops support for Python 2.7, for Python 3.7 or earlier, and
for toolkits versions before 2020.

Nearly everything else is backwards compatible with older versions of
chemfp. There are three changes which I hope breaks no existing code,
and if it does, they should be easy to fix.

1. At the API level, the default similarity search had k=3 nearest
   neighbors with threshold=0.7 as Python default parameters. This kept
   tripping me up when I because when I wanted (say) all k=10 nearest
   neigbhors I would forget to change the threshold to 0.0. The default
   now is that if neither k nor threshold are specified then do a k=3
   and threshold=0.7 search. If k is specified and threshold is not, use
   threshold=0.0. This matches the simsearch command-line behavior.

2. The ``SearchResults.reorder_all()`` ``order`` parameter has been
   renamed ‘ordering’ be consistent with ``SearchResult.reorder()`` and
   with internal use.

3. The “difference” functions in ``chemfp.bitops`` function have been
   renamed to ``xor`` because even I couldn’t remember what “difference”
   meant. The xor fuction names are ``byte_xor`` and ``hex_xor``.

Deprecation warning
-------------------

The FingeprintType method ``compute_fingerprint`` and
``compute_fingerprints`` are deprecated in favor of the have been
deprecated in favor of the names ``from_mol`` and ``from_mols``. The old
methods will be removed in a a future version of chemfp.