# SF-Matching: identification of metabolites from tandem mass spectra

SF-Matching(<u>S</u>ub<u>F</u>ragment-<u>Matching</u>) is a machine-learning based approach to predict compounds from tandem mass spectra. When using SF-Matching, please consult (and cite) the following reference:

`Identification of metabolites from tandem mass spectra with a machine learning approach utilizing structural features.`

SF-Matching can be run in different modes:

1. [**Search against pre-calculated library (recommended)**](#precalc_lib) if your compound of interest is present one of the chemical databases KEGG, HMDB, ChEBI, and ChEBML, or is a short peptide with length less than 4.
1. [**Use trained models to search other compounds**](#model1) if your compound is not contained in these chemical databases. You can download the trained models and predict if a given spectrum (in `.mgf` format) corresponds to a given molecule. 
1. [**Create you own pre-calculated library**](#model2) if you have a fixed library of chemical compounds that you want to search against. Once you have created your own pre-calculated library, you could discard the trained models and save disk space.
1. [**Train your own models**](#train) if you have access to a corpus of spectra to be used as training data.

The source code is available via the [EMBL GitLab repository](https://git.embl.de/grp-bork/sf-matching).

<a name="precalc_lib"></a>
## Search against pre-calculated library (recommended)
### Requirements
1. Python 2 or Python 3 (Python 3 is required for searching mzML file.), Python >=3.6 is recommended.
2. pandas >= 0.24.2
3. numpy >= 1.16.2


### Download link
Download the SF-Matching program and pre-calculated library using the link below:

[`sf-matching_with_precalculated_library.zip`](https://swifter.embl.de/~yli/sf_matching/download/sf-matching_with_precalculated_library.zip) (19 	GB, md5sum: ``)

The pre-calculated library will be in the folder `sf-matching/Data/library`.

You can also download the program and library from [Code Ocean](https://codeocean.com/capsule/5570439/tree) and [Zenodo](https://dx.doi.org/10.5281/zenodo.3345099). 

### Usage
1. Download the SF-Matching program with pre-calculated library and unzip it.
2. Convert your MS raw file into mgf / mzML / mzXML format.
3. Run the command:

    ```HTML
    python [path_to_the_program]/SearchWithPreCalculatedLibrary/DatabaseSearching.py \
        -db_path [path_to_library] \
        -spectra [.mgf/.mzML/.mzXML filename] \
        -out [output_filename] \
        -ion [ion_type 0 for [M-H]-, 1 for [M+H]+]
    ```

### Example
If you put the sf-matching program in `/data/sf-matching/`, the library data is in `/data/sf-matching/Data/library/`.
Now you want to search the spectrum file `/data/sf-matching/Data/examples/test-001.mgf` with [M-H]-,
the following command will generate an output file in `/data/result.tsv`.

```HTML
    python /data/sf-matching/SearchWithPreCalculatedLibrary/DatabaseSearching.py \
        -db_path /data/sf-matching/Data/library/ \
        -spectra /data/sf-matching/Data/examples/test-001.mgf \
        -out /data/result.tsv \
        -ion 0
```

    

An example is also contained in the file `example_for_score_with_precalculated_library.sh`, and on [CodeOcean](https://codeocean.com/capsule/5570439/tree).


<a name="model1"></a>
## Search with pre-built models
### Requirements
1. Python == 3.6
2. numpy == 1.16.2
3. pandas == 0.24.2
4. rdkit == 2017.09.2
5. joblib == 0.11
6. scipy == 1.2.1
7. scikit-learn == 0.20.3
8. Cython == 0.29.6

The program is tested in the version shown above, other version may work but have not been tested.

### Download link
Download the SF-Matching program and pre-built models using the link below:

[`sf-matching_with_prebuild_model.zip`](https://swifter.embl.de/~yli/sf_matching/download/sf-matching_with_prebuilt_model.zip) (106GB, md5sum:``)

The pre-built models will be in the folder `sf-matching/Data/model/neg` and `sf-matching/Data/model/pos`.


### Usage
1. Download the SF-Matching program with pre-built model and unzip it.
2. Compile the cython module with the following command:

    ```HTML
    cd [path_to_sf-matching]
    cd SearchWithModel
    python setup.py build_ext --inplace
    ```
    
3. Convert your spectrum file into mgf format, please make sure that one file only contain one spectrum.
You can find one example in the folder `sf-matching/Data/examples/test-001.mgf`
4. Prepare the candidate molecules in txt format, one line one molecule, the molecules can be in SMILES or InChI format.
You can find one example in the folder `sf-matching/Data/examples/test-001.txt`
5. Run the command:

    ```HTML
    python [path_to_the_program]/SearchWithModel/ScoreWithPreBuildModel.py \
        -spectrum [mgf_file] \
        -mol [candidate_molecule_file] \
        -model [pre_calculated_model] \
        -out_file [output_file] \
        -ion [ion_type 0 for [M-H]-, 1 for [M+H]+] \
        -threads [the_threads_you_want_to_use]
    ```

### Example
If you put the sf-matching program in `/data/sf-matching/`, the model data is in `/data/sf-matching/Data/model/neg/`.
Now you want to search the spectrum file `/data/sf-matching/Data/examples/test-001.mgf` with `[M-H]-`,
the candidate molecule is in `/data/sf-matching/Data/examples/test-001.txt`.

First, run the command below to compile a [cython](https://cython.org/) module:

```HTML
    cd /data/sf-matching/SearchWithModel/
    python setup.py build_ext --inplace
```

Then, the following command will generate an output file in `/data/result.tsv`.

```HTML
    python /data/sf-matching/SearchWithModel/ScoreWithPreBuildModel.py \
        -spectrum /data/sf-matching/Data/examples/test-001.mgf \
        -mol /data/sf-matching/Data/examples/test-001.txt \
        -model /data/sf-matching/Data/model/neg/ \
        -out_file /data/result.tsv \
        -ion 0 \
        -threads 8
```
    
If you want to search spectrum with `[M+H]+`, just change the ion to `1`, also change the model data to positive model.
Suppose this model is stored in `/data/sf-matching/Data/model/pos/`, you can use the following command:

```HTML
    python /data/sf-matching/SearchWithModel/ScoreWithPreBuildModel.py \
        -spectrum /data/sf-matching/Data/examples/test-001.mgf \
        -mol /data/sf-matching/Data/examples/test-001.txt \
        -model /data/sf-matching/Data/model/pos/ \
        -out_file /data/result.tsv \
        -ion 1 \
        -threads 8
```

Or you can find an example from the file `example_for_score_with_prebuild_model.sh`



<a name="model2"></a>
## Calculate your own library with pre-built model
### Requirements
1. Python == 3.6
2. numpy == 1.16.2
3. pandas == 0.24.2
4. rdkit == 2017.09.2
5. joblib == 0.11
6. scipy == 1.2.1
7. scikit-learn == 0.20.3
8. Cython == 0.29.6
9. sqlite3 == 3.27.2

The program is tested in the version shown above, other version may work but have not been tested.

### Download link
Download the SF-Matching program and pre-built models using the link below:

[`sf-matching_with_prebuild_model.zip`](https://swifter.embl.de/~yli/sf_matching/download/sf-matching_with_prebuild_model.zip) (106GB, md5sum:``)

The pre-built models will be in the folder `sf-matching/Data/model/neg` and `sf-matching/Data/model/pos`.


### Usage
1. Download the SF-Matching program with pre-build model and unzip it.
2. Compile the cython module with the following command:

    ```HTML
    cd [path_to_sf-matching]
    cd SearchWithModel
    python setup.py build_ext --inplace
    ```
    
3. Generate spectral library with the following command:

    ```HTML
    # Change neg to pos if you are generating [M+H]+ library
    sqlite3 [path_to_spectral_library]/molecular_spectra_neg.db < [path_to_sf-matching]/SearchWithModel/sql.txt
    ```
4. Preparing a txt file contains molecules you want to be included in the library, and run the following command to add those molecules to the spectral database:

    ```HTML
    python [path_to_sf-matching]/SearchWithModel/30_PreprocessInputForSpectrumPrediction.py \
    -in_file [file_to_molecular_inchi] \
    -db [path_to_spectral_library]/molecular_spectra_neg.db \
    -threads [processes_you_have] \
    -batch_size 10000 \ # Decrease this number if you want to seprate into more pieces in next step.
    -ion 0 # 1 for [M+H]+
    ```

5. Read the output from step 4, you will get a number for the batch_num parameter, run the command showed below. The command can be runned in different computers at the same time to shorten waiting time.

    ```HTML
    CAL_PART=[the_number_you_got_from_step_4's_result]
    for NUM in $(seq 0 1 $((${CAL_PART} - 1))); do
        python [path_to_sf-matching]/SearchWithModel/31_GenerateInSilicoSpectrum.py \
            -db [path_to_spectral_library]/molecular_spectra_neg.db \
            -spectra [path_to_temporary_folder] \
            -model [path_to_precalculated_model] \
            -batch_num ${NUM} \
            -threads [processes_you_have] \
            -ion 0 # 1 for [M+H]+
    done
    ```

6. Run the following command to get final pre-calculated library.
    ```HTML
    python [path_to_sf-matching]/SearchWithModel/32_SaveSpectrum.py \
        -db [path_to_spectral_library]/molecular_spectra_neg.db \
        -spectra [path_to_temporary_folder] \
        -model [path_to_precalculated_model] \
        -out_file [path_to_spectral_library]/spectra_data_neg.bin \
        -threads [processes_you_have] \
        -ion 0 # 1 for [M+H]+
    ```


### Example
You can find an example from the file `example_for_precalculate_library.sh`

<a name="train"></a>
## Build model from scratch
This is only for advanced user who have a database of in-house spectra that can be used for training the model. You may need some SQL knowledge.
To calculate the whole model from scratch, around 10,000 - 200,000 cpu hours are needed, depending on the size of spectral database.

### Requirement
1. Python == 3.6
2. numpy == 1.16.2
3. pandas == 0.24.2
4. rdkit == 2017.09.2
5. joblib == 0.11
6. scipy == 1.2.1
7. scikit-learn == 0.20.3
8. Cython == 0.29.6

The program is tested in the version showed above, other version may work but haven't been tested.

### Download link
Download the SF-Matching program:

[`sf-matching_with_example_spectra_library.zip`](https://swifter.embl.de/~yli/sf_matching/download/sf-matching_with_example_spectra_library.zip) (MB, md5sum:``)

The spectral database will be in the folder `sf-matching/Data/model/database`.

### Usage
1. Download the SF-Matching program with spectral database and unzip it.
2. Add your own spectral into the spectral database which is located in `sf-matching/Data/database/spectral.db`.
3. Compile the cython module with the following command:

    ```HTML
    cd [path_to_sf-matching]
    cd SearchWithModel
    python setup.py build_ext --inplace
    ```

4. Follow the example: `example_for_build_model.sh`