I’m in the process of getting the tests passing for the 0.03 release of NIST::NVD::Store::SQLite3 wherein our hero imports the CWE data and cross-indexes it with CVEs and CPEs.
Follow along and suggest some patches. I’m developing on Debian Wheezy, but I would very much like input from devs on other platforms.
http://git.colliertech.org/?p=NIST-NVD-Store-SQLite3.git;a=summary
cjac@foxtrot:/tmp$ time git clone http://git.colliertech.org/git/NIST-NVD-Store-SQLite3.git Cloning into 'NIST-NVD-Store-SQLite3'... real 0m32.757s user 0m0.200s sys 0m0.088s cjac@foxtrot:/tmp$ ls NIST-NVD-Store-SQLite3/t/data/ cwec_v2.1.xml nvdcve-2.0-test.xml
Publish your patches and I’ll fetch them, or you can submit them in udiff format and I’ll review/apply. Thanks for playing along!
[edit 20120216T1456 -0800]
Seems I need to update the NIST::NVD package as well.
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 rm: cannot remove `t/data/*.db': No such file or directory Writing Makefile for NIST::NVD::Store::SQLite3 Writing MYMETA.yml and MYMETA.json Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged) cp bin/convert-nvdcve blib/script/convert-nvdcve /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm using store [SQLite3] reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone. Writing CPE URNs to disk...Done. Writing NVD entries to disk....................................................................... Done. Writing CPE index to disk...Done. Writing CWE index to disk...Can't locate object method "put_idx_cwe" via package "NIST::NVD::Update" at /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve line 77. real 0m13.072s user 0m12.421s sys 0m0.044s
$ time git clone http://git.colliertech.org/git/NIST-NVD.git Cloning into 'NIST-NVD'... real 0m2.921s user 0m0.016s sys 0m0.024s
[edit 20120216T16:28 -0800]
cjac@foxtrot:/usr/src/git/f5/NIST-NVD$ git log | head -5 commit 82c72a79ee810c2b5c269a15dca5151ad67059f9 Author: C.J. Adams-CollierDate: Thu Feb 16 16:25:53 2012 -0800 added put_idx_cwe to NIST::NVD::Update
[edit 20120216T1635 -0800]
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 rm: cannot remove `t/data/*.db': No such file or directory Writing Makefile for NIST::NVD::Store::SQLite3 Writing MYMETA.yml and MYMETA.json Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged) cp bin/convert-nvdcve blib/script/convert-nvdcve /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm using store [SQLite3] reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................read 68 entries Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone. Writing CPE URNs to disk...Done. Writing NVD entries to disk....................................................................... Done. Writing CPE index to disk...Done. Writing CWE index to disk...Done. real 0m13.968s user 0m13.225s sys 0m0.064s
Alright. before going to bed, I made those “w” characters above mean something. Same with some of the “v” characters above.
Now processing the Categories from the CWE files. At this point we have parsed the Description elements. Next up are:
[edit 20120217T11:35 -0800]
Likelihood_of_ExploitTime_of_IntroductionAffected_ResourcesApplicable_Platforms
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ rm t/data/*.db *.db ; perl Makefile.PL ; make ; time perl -Iblib/lib /usr/src/git/f5/NIST-NVD-Store-SQLite3/blib/script/convert-nvdcve --nvd /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml --cwe /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/cwec_v2.1.xml --store SQLite3 rm: cannot remove `t/data/*.db': No such file or directory Writing Makefile for NIST::NVD::Store::SQLite3 Writing MYMETA.yml and MYMETA.json Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged) cp bin/convert-nvdcve blib/script/convert-nvdcve /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm using store [SQLite3] reading NVDs from file: /usr/src/git/f5/NIST-NVD-Store-SQLite3/t/data/nvdcve-2.0-test.xml.......................................................................Done. read 68 nvd entries Processing CWE file...vvvvvvvvvvvvvvvvvvvvvvvvvvvvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcvDDDcwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwweeeeeeeeeDone. read 27 cwe Views read 1 cwe Categories read 693 cwe Weaknesses read 1 cwe Compound Elements Writing CPE URNs to disk...Done. Writing NVD entries to disk....................................................................... Done. Writing CPE index to disk...Done. Writing CWE index to disk...Done. real 0m14.306s user 0m13.501s sys 0m0.072s
[edit 20120217T1333 -0800]
Alright, I’ve got enough of the CWE processing done that I feel comfortable releasing this stuff. Let’s make sure that the data got all the way down into the database. I’ll write some tests in t/cwe.t to exercise the CRUD
[edit 20120217T1345 -0800]
Tests are in place and failing (this is good, by the way). Now to turn the tests green.
cjac@foxtrot:/usr/src/git/f5/NIST-NVD-Store-SQLite3$ pushd ../NIST-NVD ; perl Makefile.PL ; make ; popd ; rm t/data/*.db *.db ; perl Makefile.PL ; make ; time prove -v -I../NIST-NVD/blib/lib -Iblib/lib t/cwe.t /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery Writing Makefile for NIST::NVD Writing MYMETA.yml and MYMETA.json Skip blib/lib/NIST/NVD/Store/Base.pm (unchanged) Skip blib/lib/NIST/NVD.pm (unchanged) Skip blib/lib/NIST/NVD/Query.pm (unchanged) Skip blib/lib/NIST/NVD/Update.pm (unchanged) Skip blib/lib/NIST/NVD/Store/DB_File.pm (unchanged) cp bin/convert-nvdcve blib/script/convert-nvdcve /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve Manifying blib/man3/NIST::NVD::Store::Base.3pm Manifying blib/man3/NIST::NVD.3pm Manifying blib/man3/NIST::NVD::Query.3pm Manifying blib/man3/NIST::NVD::Update.3pm Manifying blib/man3/NIST::NVD::Store::DB_File.3pm /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD /usr/src/git/f5/NIST-NVD-Store-SQLite3 /usr/src/svn/f5/axiom/branches/cjac/F5-Discovery rm: cannot remove `t/data/*.db': No such file or directory Writing Makefile for NIST::NVD::Store::SQLite3 Writing MYMETA.yml and MYMETA.json Skip blib/lib/NIST/NVD/Store/SQLite3.pm (unchanged) cp bin/convert-nvdcve blib/script/convert-nvdcve /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/convert-nvdcve Manifying blib/man3/NIST::NVD::Store::SQLite3.3pm t/cwe.t .. 1..10 ok 1 - use NIST::NVD::Query; ok 2 - no error ok 3 - constructor returned an object of correct class not ok 4 - cve_for_cpe returned ARRAY ref # Failed test 'cve_for_cpe returned ARRAY ref' # at t/cwe.t line 36. # got: '' # expected: 'ARRAY' not ok 5 - cwe_for_cpe returned ARRAY ref # Failed test 'cwe_for_cpe returned ARRAY ref' # at t/cwe.t line 37. # got: '' # expected: 'ARRAY' Can't use an undefined value as an ARRAY reference at t/cwe.t line 39. # Looks like you planned 10 tests but ran 5. # Looks like you failed 2 tests of 5 run. # Looks like your test exited with 2 just after 5. Dubious, test returned 2 (wstat 512, 0x200) Failed 7/10 subtests Test Summary Report ------------------- t/cwe.t (Wstat: 512 Tests: 5 Failed: 2) Failed tests: 4-5 Non-zero exit status: 2 Parse errors: Bad plan. You planned 10 tests but ran 5. Files=1, Tests=5, 1 wallclock secs ( 0.02 usr 0.00 sys + 0.07 cusr 0.01 csys = 0.10 CPU) Result: FAIL real 0m0.442s user 0m0.144s sys 0m0.028s
PlanetMySQL Voting: Vote UP / Vote DOWN