Jobsub ID 258467.16@dunegpschedd01.fnal.gov
| Jobsub ID | 258467.16@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 08:03:14 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce01 |
| Last heartbeat | 2025-11-26 11:30:39 |
| From worker node | Hostname | wn53 |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 08:04:23 |
| Input files | hd-protodune:pdhd_prod_beam__238205_33_1_20251106T043404Z_gen_g4_IonScintPDExt.root_256311_67_1_20251124T190033Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (12118s) |
| CPU time | 2h (9063s = 74%) |
| Max RSS bytes | 6406201344 (6109 MiB) |
| Outputting started | 2025-11-26 11:26:24 |
| Output files | |
| Finished | 2025-11-26 11:30:39 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
321 XUVs
C:0 T:7 44 XUs and 18 XVs -> 0 XUVs
12640 XUVs total
5020 collection wire objects
12640 potential space points
Neighbour search...
859194 tests to find 324130 neighbours
Iterating with no regularization...
Begin: 6.89323e+07
0 6.14982e+07
1 6.10182e+07
2 6.09791e+07
Now with regularization...
Begin: 5.86883e+07
0 5.8676e+07
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 08:26:11 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root"
26-Nov-2025 08:26:11 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 202.39 348.686 868.99 270.188 190.301 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00470682 0.0129247 0.0223128 0.0131801 0.00551913 10
reco:opdec:Deconvolution 0.934327 1.38892 1.72977 1.3683 0.23911 10
reco:ophitspe:OpHitFinderDeco 0.265378 0.397157 0.456308 0.398883 0.0489684 10
reco:opflash:OpFlashFinder 0.0846574 0.118075 0.18412 0.110567 0.0294386 10
reco:opslicer:OpSlicer 0.306801 0.433563 0.623796 0.38728 0.111423 10
reco:wclsdatahd:WireCellToolkit 76.9352 81.2786 88.4964 80.1269 3.43881 10
reco:gaushit:GausHitFinder 1.19188 1.68968 2.54398 1.60135 0.38301 10
reco:nhitsfilter:NumberOfHitsFilter 0.000442809 0.000663728 0.000971715 0.000632539 0.000147765 10
reco:reco3d:SpacePointSolver 1.30296 2.09029 4.0057 1.7079 0.958631 10
reco:hitpdune:DisambigFromSpacePoints 0.895336 2.32418 4.81748 1.91321 1.15006 10
reco:pandora:StandardPandora 28.6572 118.388 544.096 66.2416 146.897 10
reco:pandoraWriter:StandardPandora 0.305162 0.545715 0.764828 0.528863 0.136071 10
reco:pandoraTrack:LArPandoraTrackCreation 3.88277 6.90423 14.6883 5.65585 3.90229 10
reco:pandoraShower:LArPandoraShowerCreation 3.08059 5.5925 11.274 4.00895 3.00699 10
reco:pandoracalo:Calorimetry 1.70421 3.05987 6.6655 2.4688 1.677 10
reco:pandoracalonosce:Calorimetry 1.63123 3.0211 6.40482 2.38949 1.70164 10
reco:pandoraShowercalo:ShowerCalorimetry 2.80751 4.86968 9.5078 3.79577 2.26707 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.15889 4.72398 9.33383 3.55344 2.48443 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0947018 0.12183 0.14733 0.122555 0.0156911 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0966538 0.120519 0.143381 0.122031 0.014445 10
reco:emtrkmichelid:EmTrackMichelId 49.452 63.3425 84.9026 58.4 12.205 10
reco:crttag:SingleCRTMatchingProducer 0.0093224 0.0140396 0.017592 0.0142943 0.00258765 10
reco:crtreco:TwoCRTMatchingProducer 0.00786034 0.0121107 0.0160919 0.0120394 0.00222337 10
reco:pandora2Track:LArPandoraTrackCreation 7.04221 15.4433 32.5353 12.332 7.92993 10
reco:pandora2calo:ShowerCalorimetry 1.71775 4.52861 8.88567 3.43477 2.34964 10
reco:pandora2calonosce:ShowerCalorimetry 2.14788 4.50353 8.87034 3.51165 2.36443 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0971749 0.122786 0.144831 0.122631 0.0145771 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.097434 0.120203 0.141648 0.119428 0.0138536 10
reco:pandora2Shower:LArPandoraShowerCreation 3.00029 6.50558 13.5905 5.01254 3.39837 10
reco:pandora2Showercalo:ShowerCalorimetry 2.56147 6.23324 13.593 4.70847 3.51689 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.31597 6.14606 12.989 4.63701 3.31104 10
[art]:TriggerResults:TriggerResultInserter 2.8267e-05 5.13018e-05 0.000121298 4.20535e-05 2.50194e-05 10
end_path:out1:RootOutput 8.677e-06 1.22934e-05 3.3655e-05 1.00955e-05 7.17288e-06 10
end_path:out1:RootOutput(write) 3.69239 4.47449 5.85569 4.20689 0.743669 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6273.46 MB
Peak resident set size usage (VmHWM): 4077.23 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.temp.json ====
.:
total 6047736
-rw-r--r-- 1 nobody nobody 76590 Nov 26 08:26 jobscript.log
-rw-r--r-- 1 nobody nobody 203 Nov 26 08:26 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 08:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 08:26 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 08:26 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 08:26 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 08:26 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 08:26 time.db
-rw-r--r-- 1 nobody nobody 6137878226 Nov 26 08:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258467_16_1_20251126T080429Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 54639440 Nov 26 08:22 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 07:29 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 07:24 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 07:23 detsim_single_protodunehd_hist.root 