March, 2023: pdb_extract version V4.1 is now released

The followings are the improvements of this version compared to the previous version:

• Speed up mmCIF file parsing by switching from Python parser to C++ parser, except for dictionary check;
• Fix HKL-3000 log parsing error based on the latest template;
• Add Aimless log parsing from the latest autoPROC template, and combine Aimless log parsing on diffrent tempaltes into on e module;
• Convert _software to _em_software for EM entries, and classify the software based on the dominant usage of the software on PDB entries;
• Convert _software to _pdbx_nmr_software for NMR entries, classify the software based on the dominant usage of the software on PDB entries, and add software authors based on OneDep DepUI NMR software author dictionary;
• Check duplicate diffraction id and provide warning;
• On pdb_extract server, add file upload exception handling for binary files or files with non-ASCII characters;
• On pdb_extract server, add _em_software and _pdbx_nmr_software example in mmCIF-formated templates;
• On pdb_extract server, allow author to update sequence if the uploaded mmCIF-formatted file already has sequence;

November 1, 2022: pdb_extract version V4.0 is now released

The followings are the improvements of this version compared to the previous version:

• Re-design under python framework;
• Remove dependency on outdated PDB-to-CIF converter by incorporating converter utilized by wwPDB OneDep system;
• Improve mmCIF format checking and dictionary compliance;
• Collaborate with PDBj to use CIF editor for author's template file to ensure future compliance;
• Clean up dependencies on old refinement software packages;
• Simplify the features to support modern refinement and data processing packages only;

November 22, 2017: pdb_extract version V3.24 is now released

The followings are the improvements of this version compared to the previous version:

• updated new dictionary (v5)
• improved the calculation of the entity_poly groups
• fixed _refine_ls_shell.R_factor_all for new version of buster (2.11.7).
• fixed potential problem for very large MTRIX? records .
• removed potential problem for "generate/given" and matrix values extraction in extract_ncs_matrix.
• extract REMARK 230 for hybrid method, refactored write_mmcif.
• added 'THROUGHOUT' to refine.pdbx_ls_cross_valid_method if missing.
• if insertion code is @ in PDB, it is converted to '.'.
• add Source_organism_taxonomy_id to the template file.
• changed reflns_shell.number_unique_all to ?_obs.
• handle weird extract for aimless in autoPROC-pipeline by GlobalPhasing.
• corrected author name for SHARP per developer's request.
• properly handle special residue names such as "Cr","Ar", "Gr", "Ur".
• changed 'data collection' -> 'data reduction' in _software.classification.
• added XDS for data reduction.
• uncommented/added data extractions below (newly added item in dic v5).
  • _pdbx_phasing_MR.d_res_high_fit
  • _pdbx_phasing_MR.d_res_low_fit
  • _pdbx_phasing_MR.zscore_rotation
  • _pdbx_phasing_MR.LL_gain_rotation
  • _pdbx_phasing_MR.zscore_translation
  • _pdbx_phasing_MR.LL_gain_translation
  • _reflns_shell.pdbx_number_anomalous
  • _reflns.pdbx_number_anomalous

December 3, 2015: pdb_extract version V3.21 is now released

The followings are the improvements of this version compared to the previous version:

• Added functions to check the enumeration list in mmcif_pdbx_v5_next.dic Auto correction if the entered values have space or case problems. If wrong values are entered, error messages and the enumeration list from the dictionary will be given in the pdb_extract.log
• Added data extracts for some new data reduction software (such as XDS, Aimless) new data items (e.g. _reflns.pdbx_CC_half, _reflns_shell.pdbx_CC_half) are extracted from all the programs (HKL, XDS, Aimless/Scala)
• Add a new function "extract_table" to help to users generate a Table for crystallographic publication.
• Added/removed data items to comply with the D&A system requirements.
• Supports the current versions of all the external programs.
• Modified code to better identify the entity-ploy groups.
• Complete parsing the TLS groups from Buster
• Modified the template file to parse the refinement details.
• Made other extractions standard (such as the program names, versions,...)
• Simplified the compilation and installation steps
• Updated all the documentations.

August 3, 2011: pdb_extract version V3.11 is now released

The improvements of version V3.11 compared to the previous version V3.10 are the following:

• Modified extraction for REMARK 200 for hybrid methods.
• For REFMAC, change the default NCS residue range (-10 9999) to what is there in the PDB file.
• Modified extraction of COMPND/SOURCE in a PDB file.
• Modified function align_use_seqres (memory problem).
• Add three new data items:
  • _refine.pdbx_overall_SU_R_free_Cruickshank_DPI
  • _refine.pdbx_overall_SU_R_Blow_DPI
  • _refine.pdbx_overall_SU_R_free_Blow_DPI
• Removed a potential bug in parsing TLS residue range for BUSTER.
• Corrected parsing of Buster for cases like { ?|* } ?=chainID
• Corrected mis-match (PHIC and PHIC_ALL) from mtz to mmcif conversion and makestricter parsing for phic(FC with PHIC).
• Modified _software.classification.
• Corrected a potential bugs in chain_seq_info (residue ranges for TLS/NCS)

July 10, 2010: pdb_extract version V3.10 is now released

The improvements of version V3.10 compared to the previous version V3.004 are the following:

• Supports REFMAC version 5.5 and later
• Supports multiple MTZ file conversion.
• Has improved handling of NCS data items, even for cases with incomplete data.
• For PHENIX (version 1.4 & later), extracts additional items, populates TLS/NCS residue range, and automatically distinguishes between neutron and X-ray diffraction data.
• Completely parses RMSD for BUSTER-TNT (version 2.8 & later).
• Extracts TLS residue ranges for BUSTER-TNT (version 2.8 & later).
• Improved parsing of MLPHARE log files.
• Automatically detects identity type and assigns DNA (_entity_poly.type) as polydeoxyribonucleotide and RNA (_entity_poly.type) as polyribonucleotide.
• Populates the complete coordinate file according to the current mmCIF dictionary.
• Supports data extraction for MOLPROBITY.
• Provides more descriptive error messages.
• Has updated contact information (e.g. email address, authors, URL, ...) for all supported programs.
• Automatically removes unit for temperatures

Sep 17, 2007: pdb_extract version V3.004 is now released

The improvements of version V3.004 compared to the previous version V2.00 are the following:

• Added support for BP3 (version 1.0) for heavy atom phasing refinement.
• Added support for X-gen (versions 5.5.5 and 5.8.3) for processing of diffraction data.
• Added support for COMO (version 1.2) for molecular replacement.
• Added support for BEAST (version 1.1.1) for molecular replacement.
• Added support for Xplor-NIH (version 2.13) for NMR structure refinement.)
• Added the NCS extraction from the log file of program CNS (V1.2)
• Added statistics extraction from the summary file (in addition to the log file) of the program PHASER (V2.1)
• Added an option (-sol) in the EXTRACT program to update the Matthew coefficient and solvent constant once given the correct sequence in the template file.
• Added an option (-chain) to add chain to PDB file if missing.
• Fixed a bug for data extraction regarding to the anisou record.
• The data template file now fully corresponds to the ADIT data items.
• Support for PDB v3.1 nomenclature of atoms and residues.
• Updated all the Xray program extractions for experiments [MR, SAD, MAD, SIR, SIRAS, MIR, MIRAS, AB_INITIO]
• Added more error/warning messages in the program.
• Updated the core mmcif modules to the latest version with the STL feature.
Supported crystallographic software lists     TOP

Software applications supported by pdb_extract are listed in the Table below.

Category	Software	Versions	References
Data collection integration reduction scaling averaging	HKL/HKL2000 SCALEPACK/DENZO	1.30 , 1.96 , 1.97.9	Otwinowski & Minor (1997)
	D*trek	7.0SSI , 7.11 , 9.2 , 9.7	Pflugrath (1997)
	SCALA (CCP4)	CCP4(v4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	Evans (1997)
	XDS/XSCALE	Nov. 2005 , June 2006, Dec. 2006 , Mar. 2007 , July 2007	Kabsch
	MOSFLM	6.2.2 , 6.2.3, 6.2.5, 7.0.1	Leslie(1998)
	X-gen	5.5.5 , 5.8.3	Andrew J. Howard
	SAINT	V6.35A, V7.03A	Bruker (2002)
	3DSCALE/PROSCALE	N/A	Fu (2005)
Molecular replacement	CNS/CNX	0.9 , 1.0, 1.1 , 1.2	Brunger et al. (1998)
	XPLOR	3.1 , 3.851	Brunger et al (1998)
	AMORE (CCP4)	CCP4(V4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	Navaza (1994)
	MOLREP (CCP4)	CCP4(V4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	Vagin & Teplyakov (1997)
	EPMR	2.5	Kissinger et al. (1999)
	PHASER	1.2, 1.3 , 2.0 , 2.1	Read(2001)
	BEAST	1.1.1	Read(2001)
	COMO	1.2	Tong(1996)
Heavy atom phase determination	CNS/CNX	0.9 , 1.0 , 1.1 , 1.2	Brunger et al. (1998)
	XPLOR	3.1 , 3.851	Brunger et al (1998)
	SOLVE	2.0 , 2.01, 2.02, 2.06, 2.08 , 2.09, 2.10, 2.11, 2.13	Terwilliger & Berendzen. (1999)
	MLPHARE (CCP4)	CCP4(V4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	CCP4 (1994)
	SHARP/autoSHARP	1.3.x , 1.4.0 , 2.0 , 2.01 , 2.04 , 2.2.0	Fortelle & Bricogne (1997)
	SHELXD/SHELXS	97	Sheldrick (1997)
	PHASES	95	Furey (1997)
	PHASER	2.0 , 2.1	Read(2001)
	SnB	2.0 , 2.1 , 2.2	Weeks & Miller (1999)
	BnP	0.93 , 1.0 , 1.02 , 1.05	Weeks et al. (2002)
	BP3	1.0	Navraj S. Pannu(2003)
Density modification	CNS/CNX	0.9 , 1.0 , 1.1 , 1.2	Brunger et al. (1998)
	XPLOR	3.1 , 3.851	Brunger et al (1998)
	DM (CCP4)	CCP4(v4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	Cowtan (1994)
	SOLOMON (CCP4)	CCP4(V4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	Abrahams & Leslie (1996)
	RESOLVE	2.0 , 2.01, 2.02, 2.06, 2.08 , 2.09, 2.10, 2.11, 2.13	Terwilliger (2000)
	SHELXE	97	Sheldrick (1997)
Structure refinement	CNS/CNX	0.9 , 1.0 , 1.1 , 1.2	Brunger et al. (1998)
	XPLOR	3.1 , 3.851	Brunger et al (1998)
	REFMAC5 (CCP4)	CCP4(V4.0 , 5.0 , 5.01, 5.02 , 6.0 , 6.01 , 6.02)	Murshudov (1997)
	PHENIX	1.0 , 1.1a , 1.22a , 1.3.1 , 1.3b	Adams et al (2002)
	SHELXL	97	Sheldrick (1997)
	TNT	5F	Tronrud (1997)
	BUSTER-TNT	1.0.2	G.Bricogne (1993)
	ARP/wARP	5.0 , 6.1.1 , 7.0	Lamzin & Wilson, (1997)
	RESTRAIN (CCP4)	4.6	CCP4 (1994)
NMR structure determination	CNS/CNX	1.1 , 1.2	Brunger et al. (1998)
	XPLOR	3.1 , 3.851	Brunger et al (1998)
	CYANA	2.0	Güntert (1997)
	Xplor-NIH	2.13	G. Marius Clore(2003)

References     TOP
1. Z. Otwinowski and W. Minor. (1997). Processing of X-ray Diffraction Data Collected in Oscillation Mode. Methods in Enzymology, Volume 276: Macromolecular Crystallography, part A, p.307- 326
2. Pflugrath JW (1999). The finer things in X-ray diffraction data collection. Acta Cryst. D55 1718-25
3. Zheng-Qing Fu (2005), Three-dimensional model-free experimental error correction of protein crystal diffraction data with free-R test Acta Cryst. D61 1643-1648
4. SAINT V6.35A, Bruker Analytical X-Ray Systems, Madison, WI, (2002).
5. Evens, P. R. (1997). "the Scala" Joint CCP4 and ESF-EACBM Newsletter. 33, 22-24
6. Kabsch, W. (1993). Automatic processing of rotation diffraction data from crystals of initially unknown symmetry and cell constants. J. Appl. Cryst. 26, 795-800.
7. Leslie A. G. W. (1998), J. Appl. Cryst. 30, 1036-1040.
8. Brunger, A.T., Adams, P.D., Clore, G.M., DeLano, W.L., Gros, P., Grosse-Kunstleve, R.W., Jiang, J.-S., Kuszewski, J., Nilges, N., Pannu, N.S., Read, R.J., Rice, L.M., Simonson, T., and Warren, G.L. (1998). Crystallography and NMR system (CNS): A new software system for macromolecular structure determination. Acta Cryst. D54, 905-921.
9. Navaza J. (1994) AMoRe: an Automated Package-- --for Molecular Replacement. Acta Cryst. D50, 157-163.
10. Vagin A. , Teplyakov A. (1997) , MOLREP: an automated program for molecular replacement. J. Appl. Cryst. 30, 1022-1025.
11. Charles R. Kissinger, Daniel K. Gehlhaar & David B. Fogel, (1999) Rapid automated molecular replacement by evolutionary search. Acta Cryst. , D55, 484-491
12. R. J. Read (2001) Pushing the boundaries of molecular replacement with maximum likelihood. Acta Cryst. D57, 1373-1382
13. Terwilliger, T.C. and J. Berendzen. (1999) Automated MAD and MIR structure solution. Acta Cryst. D55, 849-861.
14. COLLABORATIVE COMPUTATIONAL PROJECT, NUMBER 4. 1994. The CCP4 Suite: Programs for Protein Crystallography. Acta Cryst. D50, 760-763
15. E. de La Fortelle & G. Bricogne (1997) Maximum-Likelihood Heavy-Atom Parameter Refinement for the Multiple Isomorphous Replacement and Multiwavelength Anomalous Diffraction Methods. Methods in Enzymology 276 472-494
16. Furey, W. & Swaminathan, S. (1997), PHASES-95: A Program Package for the Processing and Analysis of Diffraction Data from Macromolecules. Methods in Enzymology, 277, 590-620
17. Weeks, C.M. & Miller, R. (1999). The design and implementation of SnB v2.0, J. Appl. Cryst.32, 120-124.
18. Weeks, C.M., Blessing, R.H., Miller, R., Mungee, S., Potter, Rappleye, A., Simith, G.D. Xu, H., Furey, W. (2002), Towards automated protein structure determination: BnP, the SnB-PHASES Interface. Z. Kristallogr. 217, 686-693
19. Navraj S. Pannu,Airlie J. McCoy, Randy J. Read(2003), Application of the-- --complex multivariate normal distribution to crystallographic methods with insights into multiple isomorphous replacement phasing ACTA CRYSTALLOGR.,SECT.D. 59, 1801-1808
20. Sheldrick G. (1997) The SHELX-97 homepage http://shelx.uni-ac.gwdg.de/SHELX/
21. K. Cowtan (1994), Joint CCP4 and ESF-EACBM Newsletter on Protein Crystallography. 31, p34- 38.
22. Abrahams J. P. and Leslie A. G. W.(1996). Acta Cryst. D52, 30-42
23. Terwilliger, T. C. (2000) Maximum likelihood-- --density modification. Acta Cryst. D56, 965-972.
24. G. Bricogne (1993), Direct Phase-- --Determination by Entropy Maximisation and Likelihood Ranking: Status Report and Perspectives. ACTA CRYSTALLOGR.,SECT.D 49, 37-60
25. Tronrud, D, E., (1997). The TNT Refinement Package. in Macromolecular Crystallography, Part B, Methods Enzymol. 277, 306-318
26. Lamzin, V.S. & Wilson, K.S. (1997). Automated refinement for protein crystallography. Methods Enzymol. (Carter, C. & Sweet, B. eds.) 277, 269-305
27. G.N. Murshudov, A.A.Vagin and E.J.Dodson, (1997) Refinement of Macromolecular Structures by the Maximum-Likelihood Method. Acta Cryst. D53, 240-255.
28. P.D. Adams, R.W. Grosse-Kunstleve,-- --L.-W. Hung, T.R. Ioerger, A.J. McCoy, N.W. Moriarty, R.J. Read, J.C. Sacchettini, N.K. Sauter and T.C. Terwilliger.(2002) PHENIX: building new software for automated crystallographic structure determination. Acta Cryst. D58, 1948-1954
29. Güntert, P., Mumenthaler, C. & Wüthrich, K. (1997). Torsion angle dynamics for NMR structure calculation with the new program DYANA. J. Mol. Biol. 273, 283-298.
30. C.D. Schwieters, J.J. Kuszewski, N. Tjandra and G.M. Clore (2003), "The Xplor-NIH NMR Molecular Structure Determination Package," J. Magn. Res. 160, 66-74.