Jobsub ID 36358.61@dunegpschedd02.fnal.gov
| Jobsub ID | 36358.61@dunegpschedd02.fnal.gov |
| Workflow ID | 2329 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:29:56 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2025-09-08 15:52:36 |
| From worker node | Hostname | wn-snel-025.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7H12 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:34:56 |
| Input files | vd-protodune:np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4163s) |
| CPU time | 50m (3056s = 73%) |
| Max RSS bytes | 2601885696 (2481 MiB) |
| Outputting started | 2025-09-08 15:44:20 |
| Output files | |
| Finished | 2025-09-08 15:52:36 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.82 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.72 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.44 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Aux::Resampler : 0.28 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Aux::Resampler : 0.28 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Aux::Resampler : 0.28 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Aux::Resampler : 0.28 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:41:09.920] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:41:09.920] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:41:09.920] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:41:09.920] I [ timer ] Timer: Total node execution : 42.32999981380999 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 41138 traces tagged "gauss"
FrameSaver: q=9.40532e+06 n=935901 tag=gauss
wclsFrameSaver: saving 51121 traces tagged "wiener"
FrameSaver: q=1.01161e+07 n=902527 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1065 XUs and 1427 XVs -> 43 XUVs
C:0 T:1 7431 XUs and 13468 XVs -> 646 XUVs
C:0 T:2 201 XUs and 471 XVs -> 10 XUVs
C:0 T:3 513 XUs and 410 XVs -> 59 XUVs
C:0 T:4 1524 XUs and 2231 XVs -> 64 XUVs
C:0 T:5 298 XUs and 327 XVs -> 23 XUVs
C:0 T:6 276 XUs and 198 XVs -> 21 XUVs
C:0 T:7 456 XUs and 671 XVs -> 32 XUVs
C:0 T:8 11399 XUs and 35725 XVs -> 4788 XUVs
C:0 T:9 3823 XUs and 5734 XVs -> 831 XUVs
C:0 T:10 3149 XUs and 5075 XVs -> 1138 XUVs
C:0 T:11 1382 XUs and 1494 XVs -> 59 XUVs
C:0 T:12 5331 XUs and 6431 XVs -> 505 XUVs
C:0 T:13 1970 XUs and 2904 XVs -> 101 XUVs
C:0 T:14 396 XUs and 458 XVs -> 35 XUVs
C:0 T:15 3318 XUs and 4747 XVs -> 431 XUVs
8786 XUVs total
1753 collection wire objects
8786 potential space points
Neighbour search...
1094350 tests to find 461992 neighbours
Iterating with no regularization...
Begin: 1.21803e+10
0 1.18457e+10
1 1.18262e+10
2 1.18253e+10
Now with regularization...
Begin: 1.17e+10
0 1.16994e+10
PandoraContentApi::GetList(*this, m_inputClusterListName, pInputClusterList) throw STATUS_CODE_NOT_INITIALIZED
in function: CollectThirdViewClusters
in file: /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArThreeDReco/LArCosmicRay/TwoViewDeltaRayMatchingAlgorithm.cc line#: 174
Failure in algorithm Alg0075, LArTwoViewDeltaRayMatching, STATUS_CODE_NOT_INITIALIZED
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 17:42:14 CEST Closed output file "np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234_reco_stage1_20250908T154214_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 81.3817 142.583 265.08 136.751 34.1809 28
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.825e-05 7.77475e-05 0.000233007 7.17545e-05 3.08384e-05 28
produce:tpcrawdecoder:PDVDTPCReader 18.6117 36.4376 60.3657 36.6963 11.1515 28
produce:triggerrawdecoder:PDVDTriggerReader4 0.0347671 0.0364148 0.0502397 0.0351362 0.00374645 28
produce:pdvddaphne:DAPHNEReaderPDVD 4.00912 5.89468 8.50303 5.32204 1.38557 28
produce:ophit:OpHitFinder 0.0394856 0.0550314 0.0681559 0.0566516 0.00721071 28
produce:opflash:OpFlashFinderVerticalDrift 0.00835575 0.0172435 0.0440205 0.0162165 0.0062316 28
produce:wclsdatavd:WireCellToolkit 35.5915 42.0162 48.0304 42.3993 2.55996 28
produce:gaushit:GausHitFinder 0.578982 1.0471 1.79072 1.04886 0.239494 28
produce:nhitsfilter:NumberOfHitsFilter 0.000154109 0.000287083 0.00046811 0.000276174 7.3178e-05 28
produce:reco3d:SpacePointSolver 4.46076 10.1256 19.6331 9.75224 3.00528 28
produce:hitpdune:DisambigFromSpacePoints 0.0655971 0.151966 0.317068 0.147607 0.0557824 28
produce:pandora:StandardPandora 11.5529 42.9121 134.403 36.5774 25.9063 28
produce:pandoraTrack:LArPandoraTrackCreation 0.177273 0.562449 1.13439 0.475821 0.255613 28
produce:pandoraGnocalo:GnocchiCalorimetry 0.00985089 0.0222102 0.0329048 0.0220451 0.00533705 28
[art]:TriggerResults:TriggerResultInserter 1.602e-05 2.12881e-05 4.6968e-05 1.8114e-05 6.5982e-06 28
end_path:out1:RootOutput 3.277e-06 4.42121e-06 1.8094e-05 3.8525e-06 2.67268e-06 28
end_path:out1:RootOutput(write) 2.98445 3.24993 3.55412 3.27053 0.140675 28
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4479.59 MB
Peak resident set size usage (VmHWM): 2601.89 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234_reco_stage1_20250908T154214_keepup.root
\tHists: np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234_reco_stage1_20250908T154215_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234_reco_stage1_20250908T154214_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234_reco_stage1_20250908T154214_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346135.0,
"core.end_time": 1757346135.0,
"core.events": [
1469169,
1469189,
1469209,
1469229,
1469249,
1469269,
1469289,
1469309,
1469329,
1469349,
1469369,
1469389,
1469409,
1469429,
1469449,
1469469,
1469489,
1469509,
1469529,
1469549,
1469569,
1469589,
1469609,
1469629,
1469649,
1469669,
1469689,
1469709
],
"core.event_count": 28,
"core.first_event_number": 1469169,
"core.last_event_number": 1469709,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234_reco_stage1_20250908T154215_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346135.0,
"core.end_time": 1757346135.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2798_df-s03-d3_dw_0_20250908T023234.hdf5"
}
]
} 