Jobsub ID 36356.54@dunegpschedd02.fnal.gov
| Jobsub ID | 36356.54@dunegpschedd02.fnal.gov |
| Workflow ID | 2332 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:25:57 |
| Site | CZ_FZU |
| Entry | Nova_CZ_FZU_osgce2 |
| Last heartbeat | 2025-09-08 15:52:38 |
| From worker node | Hostname | milan23.farm.particle.cz |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:28:52 |
| Input files | vd-protodune:np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4801s) |
| CPU time | 1h (4121s = 85%) |
| Max RSS bytes | 2833960960 (2702 MiB) |
| Outputting started | 2025-09-08 15:48:55 |
| Output files | |
| Finished | 2025-09-08 15:52:38 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
] <Pgrapher:> graph execution complete
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.4 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.37 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.38 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.12 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.08 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.95 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.64 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.3 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Aux::Resampler : 0.59 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Aux::Resampler : 0.48 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Aux::Resampler : 0.41 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Aux::Resampler : 0.39 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:43:10.946] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:43:10.946] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:43:10.946] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:43:10.946] I [ timer ] Timer: Total node execution : 54.14999923855066 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 33661 traces tagged "gauss"
FrameSaver: q=6.20311e+06 n=729317 tag=gauss
wclsFrameSaver: saving 41872 traces tagged "wiener"
FrameSaver: q=6.69591e+06 n=697102 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 621 XUs and 856 XVs -> 29 XUVs
C:0 T:1 157 XUs and 282 XVs -> 2 XUVs
C:0 T:2 377 XUs and 538 XVs -> 24 XUVs
C:0 T:3 40 XUs and 41 XVs -> 1 XUVs
C:0 T:4 107 XUs and 138 XVs -> 7 XUVs
C:0 T:5 261 XUs and 232 XVs -> 9 XUVs
C:0 T:6 216 XUs and 433 XVs -> 16 XUVs
C:0 T:7 122 XUs and 252 XVs -> 10 XUVs
C:0 T:8 1506 XUs and 1326 XVs -> 104 XUVs
C:0 T:9 1564 XUs and 2576 XVs -> 317 XUVs
C:0 T:10 4617 XUs and 7114 XVs -> 1070 XUVs
C:0 T:11 11166 XUs and 26408 XVs -> 3336 XUVs
C:0 T:12 17883 XUs and 45533 XVs -> 7667 XUVs
C:0 T:13 1938 XUs and 4391 XVs -> 139 XUVs
C:0 T:14 1243 XUs and 925 XVs -> 125 XUVs
C:0 T:15 555 XUs and 423 XVs -> 30 XUVs
12886 XUVs total
1575 collection wire objects
12886 potential space points
Neighbour search...
2234898 tests to find 873988 neighbours
Iterating with no regularization...
Begin: 4.18726e+08
0 3.47725e+08
1 3.4502e+08
2 3.44673e+08
3 3.44558e+08
Now with regularization...
Begin: 3.35089e+08
0 3.34962e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 17:44:26 CEST Closed output file "np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726_reco_stage1_20250908T154426_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 89.323 149.438 276.419 140.412 47.6502 30
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.26e-05 0.000107652 0.000291789 9.5745e-05 4.35076e-05 30
produce:tpcrawdecoder:PDVDTPCReader 11.2971 12.6651 17.5509 12.0123 1.65074 30
produce:triggerrawdecoder:PDVDTriggerReader4 0.471984 0.483775 0.53506 0.480841 0.0120513 30
produce:pdvddaphne:DAPHNEReaderPDVD 0.000405579 0.000515034 0.000822639 0.000496599 8.52843e-05 30
produce:ophit:OpHitFinder 0.000147799 0.000293283 0.00102762 0.000268009 0.000149165 30
produce:opflash:OpFlashFinderVerticalDrift 5.693e-05 9.69669e-05 0.00039825 8.89395e-05 5.86581e-05 30
produce:wclsdatavd:WireCellToolkit 40.1185 55.3391 102.31 53.1295 12.7964 30
produce:gaushit:GausHitFinder 0.95549 1.48064 2.75485 1.47648 0.413558 30
produce:nhitsfilter:NumberOfHitsFilter 0.000218669 0.000341159 0.000736919 0.000318634 0.000115184 30
produce:reco3d:SpacePointSolver 5.33622 12.2797 22.2845 12.3801 3.99991 30
produce:hitpdune:DisambigFromSpacePoints 0.075416 0.21259 0.405551 0.197887 0.0916828 30
produce:pandora:StandardPandora 18.3833 61.4993 136.773 56.1542 34.6247 30
produce:pandoraTrack:LArPandoraTrackCreation 0.255859 0.791117 2.03291 0.648719 0.446967 30
produce:pandoraGnocalo:GnocchiCalorimetry 0.0117871 0.0311224 0.0528292 0.0318007 0.0110194 30
[art]:TriggerResults:TriggerResultInserter 1.384e-05 3.18499e-05 0.00010223 2.9e-05 1.72129e-05 30
end_path:out1:RootOutput 4.43e-06 1.13263e-05 3.341e-05 1.0605e-05 5.82507e-06 30
end_path:out1:RootOutput(write) 3.86656 4.60025 7.31222 4.44202 0.739558 30
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4750.39 MB
Peak resident set size usage (VmHWM): 2833.96 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726_reco_stage1_20250908T154426_keepup.root
\tHists: np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726_reco_stage1_20250908T154426_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726_reco_stage1_20250908T154426_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726_reco_stage1_20250908T154426_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346267.0,
"core.end_time": 1757346267.0,
"core.events": [
3549,
3569,
3589,
3609,
3629,
3649,
3669,
3689,
3709,
3729,
3749,
3769,
3789,
3809,
3829,
3849,
3869,
3889,
3909,
3929,
3949,
3969,
3989,
4009,
4029,
4049,
4069,
4089,
4109,
4129
],
"core.event_count": 30,
"core.first_event_number": 3549,
"core.last_event_number": 4129,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39338
],
"core.runs_subruns": [
3933800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726_reco_stage1_20250908T154426_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346267.0,
"core.end_time": 1757346267.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39338
],
"core.runs_subruns": [
3933800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039338_0006_df-s03-d3_dw_0_20250908T113726.hdf5"
}
]
} 