Jobsub ID 36357.81@dunegpschedd02.fnal.gov
| Jobsub ID | 36357.81@dunegpschedd02.fnal.gov |
| Workflow ID | 2332 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:27:58 |
| Site | CZ_FZU |
| Entry | Nova_CZ_FZU_osgce1 |
| Last heartbeat | 2025-09-08 15:52:28 |
| From worker node | Hostname | milan19.farm.particle.cz |
| cpuinfo | AMD EPYC 7543 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 172800 (48 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:32:21 |
| Input files | vd-protodune:np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4452s) |
| CPU time | 1h (3813s = 85%) |
| Max RSS bytes | 2993991680 (2855 MiB) |
| Outputting started | 2025-09-08 15:46:34 |
| Output files | |
| Finished | 2025-09-08 15:52:28 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
de0sigproc0> EOS at call=57 anode=0
[17:40:59.439] D [ glue ] <FrameFanin:nfsp> EOS at call=57 with 8
[17:40:59.439] D [ glue ] frame sink sees EOS
[17:40:59.439] D [ pgraph ] <Pgrapher:> graph execution complete
[17:40:59.439] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.74 sec
[17:40:59.439] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.01 sec
[17:40:59.439] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.9 sec
[17:40:59.439] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.59 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.32 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.09 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.02 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.82 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Aux::Resampler : 0.38 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Aux::Resampler : 0.38 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Aux::Resampler : 0.37 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Aux::Resampler : 0.32 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[17:40:59.440] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:40:59.440] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[17:40:59.440] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[17:40:59.440] I [ timer ] Timer: Total node execution : 51.950000712648034 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 41699 traces tagged "gauss"
FrameSaver: q=1.35085e+07 n=1124562 tag=gauss
wclsFrameSaver: saving 52283 traces tagged "wiener"
FrameSaver: q=1.40224e+07 n=1079877 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 376 XUs and 316 XVs -> 12 XUVs
C:0 T:1 1124 XUs and 815 XVs -> 63 XUVs
C:0 T:2 305 XUs and 305 XVs -> 16 XUVs
C:0 T:3 1426 XUs and 1830 XVs -> 113 XUVs
C:0 T:4 259 XUs and 222 XVs -> 15 XUVs
C:0 T:5 33 XUs and 26 XVs -> 0 XUVs
C:0 T:6 428 XUs and 363 XVs -> 13 XUVs
C:0 T:7 864 XUs and 818 XVs -> 46 XUVs
C:0 T:8 13177 XUs and 21687 XVs -> 3669 XUVs
C:0 T:9 1193 XUs and 2409 XVs -> 443 XUVs
C:0 T:10 1502 XUs and 2953 XVs -> 66 XUVs
C:0 T:11 423 XUs and 1014 XVs -> 38 XUVs
C:0 T:12 20510 XUs and 53876 XVs -> 1612 XUVs
C:0 T:13 1273 XUs and 2428 XVs -> 163 XUVs
C:0 T:14 2224 XUs and 3265 XVs -> 244 XUVs
C:0 T:15 4611 XUs and 6563 XVs -> 719 XUVs
7232 XUVs total
2373 collection wire objects
7232 potential space points
Neighbour search...
703438 tests to find 314702 neighbours
Iterating with no regularization...
Begin: 1.30355e+10
0 1.25564e+10
1 1.25331e+10
2 1.25327e+10
Now with regularization...
Begin: 1.25003e+10
0 1.25001e+10
08-Sep-2025 17:42:49 CEST Closed output file "np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525_reco_stage1_20250908T154249_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 71.6976 144.163 392.878 130.392 62.1098 29
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.939e-05 0.000102592 0.000241299 8.996e-05 3.52133e-05 29
produce:tpcrawdecoder:PDVDTPCReader 10.1883 14.5609 52.6363 11.2076 8.48322 29
produce:triggerrawdecoder:PDVDTriggerReader4 0.470479 0.483842 0.549325 0.477429 0.0167329 29
produce:pdvddaphne:DAPHNEReaderPDVD 0.000282558 0.000481251 0.00103217 0.000452587 0.000131219 29
produce:ophit:OpHitFinder 0.000179849 0.000425703 0.00148229 0.000272049 0.000365669 29
produce:opflash:OpFlashFinderVerticalDrift 5.093e-05 8.66155e-05 0.000421078 7.395e-05 6.52915e-05 29
produce:wclsdatavd:WireCellToolkit 36.7066 49.5981 86.2032 45.5987 12.1075 29
produce:gaushit:GausHitFinder 0.488264 1.14045 2.28492 1.13376 0.4362 29
produce:nhitsfilter:NumberOfHitsFilter 0.000157959 0.000495695 0.000807085 0.000499088 0.000146088 29
produce:reco3d:SpacePointSolver 3.85526 11.3914 20.6481 10.5989 3.9281 29
produce:hitpdune:DisambigFromSpacePoints 0.0471573 0.167041 0.30901 0.162401 0.0635492 29
produce:pandora:StandardPandora 12.5282 62.2646 300.357 49.3344 54.7048 29
produce:pandoraTrack:LArPandoraTrackCreation 0.18991 0.609039 1.39487 0.619089 0.254375 29
produce:pandoraGnocalo:GnocchiCalorimetry 0.0135098 0.0258752 0.0470196 0.0243342 0.00807592 29
[art]:TriggerResults:TriggerResultInserter 1.497e-05 2.61782e-05 6.266e-05 2.505e-05 8.4772e-06 29
end_path:out1:RootOutput 4.17e-06 9.17272e-06 3.3829e-05 8.26e-06 4.98864e-06 29
end_path:out1:RootOutput(write) 2.93004 3.87855 6.03921 3.72334 0.692475 29
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4930.54 MB
Peak resident set size usage (VmHWM): 2993.99 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525_reco_stage1_20250908T154249_keepup.root
\tHists: np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525_reco_stage1_20250908T154250_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525_reco_stage1_20250908T154249_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525_reco_stage1_20250908T154249_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346170.0,
"core.end_time": 1757346170.0,
"core.events": [
2408,
2428,
2448,
2468,
2488,
2508,
2528,
2548,
2568,
2588,
2608,
2628,
2648,
2668,
2688,
2708,
2728,
2748,
2768,
2788,
2808,
2828,
2848,
2868,
2888,
2908,
2928,
2948,
2968
],
"core.event_count": 29,
"core.first_event_number": 2408,
"core.last_event_number": 2968,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39338
],
"core.runs_subruns": [
3933800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525_reco_stage1_20250908T154250_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346170.0,
"core.end_time": 1757346170.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39338
],
"core.runs_subruns": [
3933800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039338_0004_df-s03-d2_dw_0_20250908T113525.hdf5"
}
]
} 