Jobsub ID 36363.141@dunegpschedd02.fnal.gov
| Jobsub ID | 36363.141@dunegpschedd02.fnal.gov |
| Workflow ID | 2326 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:51:55 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce02 |
| Last heartbeat | 2025-09-08 15:52:44 |
| From worker node | Hostname | dune001-8038717.0-lcg2725.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:46:10 |
| Input files | vd-protodune:np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (3769s) |
| CPU time | 52m (3140s = 83%) |
| Max RSS bytes | 2883223552 (2749 MiB) |
| Outputting started | 2025-09-08 15:49:00 |
| Output files | |
| Finished | 2025-09-08 15:52:44 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
at call=51 with 8
[15:43:02.763] D [ glue ] frame sink sees EOS
[15:43:02.763] D [ pgraph ] <Pgrapher:> graph execution complete
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.01 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.89 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.61 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.34 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.72 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.34 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 4.07 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 3.94 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Aux::Resampler : 0.34 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Aux::Resampler : 0.33 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Aux::Resampler : 0.3 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Aux::Resampler : 0.29 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:02.763] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:02.763] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[15:43:02.763] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[15:43:02.763] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[15:43:02.763] I [ timer ] Timer: Total node execution : 45.270000571385026 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 43664 traces tagged "gauss"
FrameSaver: q=9.10387e+06 n=978085 tag=gauss
wclsFrameSaver: saving 53501 traces tagged "wiener"
FrameSaver: q=9.70042e+06 n=940120 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 913 XUs and 1550 XVs -> 58 XUVs
C:0 T:1 1098 XUs and 1266 XVs -> 46 XUVs
C:0 T:2 887 XUs and 576 XVs -> 34 XUVs
C:0 T:3 580 XUs and 499 XVs -> 27 XUVs
C:0 T:4 1627 XUs and 2103 XVs -> 73 XUVs
C:0 T:5 490 XUs and 527 XVs -> 13 XUVs
C:0 T:6 649 XUs and 626 XVs -> 27 XUVs
C:0 T:7 395 XUs and 751 XVs -> 40 XUVs
C:0 T:8 29425 XUs and 72726 XVs -> 8295 XUVs
C:0 T:9 3818 XUs and 5184 XVs -> 393 XUVs
C:0 T:10 1370 XUs and 1199 XVs -> 74 XUVs
C:0 T:11 2358 XUs and 2424 XVs -> 127 XUVs
C:0 T:12 1736 XUs and 1768 XVs -> 77 XUVs
C:0 T:13 1452 XUs and 1201 XVs -> 101 XUVs
C:0 T:14 1464 XUs and 1396 XVs -> 81 XUVs
C:0 T:15 528 XUs and 458 XVs -> 21 XUVs
9487 XUVs total
1632 collection wire objects
9487 potential space points
Neighbour search...
1520437 tests to find 553540 neighbours
Iterating with no regularization...
Begin: 8.20448e+08
0 7.10337e+08
1 6.98964e+08
2 6.97034e+08
3 6.96273e+08
4 6.95883e+08
Now with regularization...
Begin: 6.86158e+08
0 6.85799e+08
08-Sep-2025 15:44:35 UTC Closed output file "np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110_reco_stage1_20250908T154435_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 82.6295 132.725 316.641 113.419 51.635 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.3401e-05 0.000116073 0.000269257 9.97105e-05 4.63471e-05 26
produce:tpcrawdecoder:PDVDTPCReader 13.5396 14.6406 20.3853 13.8969 2.04509 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.0461529 0.0468224 0.0484589 0.0466964 0.000510533 26
produce:pdvddaphne:DAPHNEReaderPDVD 2.75091 3.2659 4.11541 3.23377 0.317311 26
produce:ophit:OpHitFinder 0.0315312 0.045689 0.0536045 0.0462672 0.00521892 26
produce:opflash:OpFlashFinderVerticalDrift 0.0048714 0.0118618 0.021682 0.0109927 0.00401808 26
produce:wclsdatavd:WireCellToolkit 33.9895 44.5019 68.6821 43.725 7.9403 26
produce:gaushit:GausHitFinder 0.54515 1.17955 1.7704 1.20096 0.323834 26
produce:nhitsfilter:NumberOfHitsFilter 0.000133452 0.000408474 0.00100678 0.000345708 0.000246565 26
produce:reco3d:SpacePointSolver 6.05087 11.6298 20.7933 11.0891 4.0304 26
produce:hitpdune:DisambigFromSpacePoints 0.088626 0.19374 0.438166 0.165812 0.0892077 26
produce:pandora:StandardPandora 19.8423 52.208 222.705 38.2092 44.0167 26
produce:pandoraTrack:LArPandoraTrackCreation 0.224361 0.774809 1.62032 0.7261 0.434533 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.00951762 0.0288238 0.0544113 0.0293962 0.0122305 26
[art]:TriggerResults:TriggerResultInserter 2.3025e-05 4.25743e-05 0.000113151 3.989e-05 1.74874e-05 26
end_path:out1:RootOutput 3.996e-06 1.20623e-05 3.7327e-05 1.1302e-05 7.76117e-06 26
end_path:out1:RootOutput(write) 2.87449 4.12422 6.24743 4.0996 0.761093 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5047.72 MB
Peak resident set size usage (VmHWM): 2883.22 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110_reco_stage1_20250908T154435_keepup.root
\tHists: np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110_reco_stage1_20250908T154435_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110_reco_stage1_20250908T154435_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110_reco_stage1_20250908T154435_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346276.0,
"core.end_time": 1757346276.0,
"core.events": [
1171109,
1171129,
1171149,
1171169,
1171189,
1171209,
1171229,
1171249,
1171269,
1171289,
1171309,
1171329,
1171349,
1171369,
1171389,
1171409,
1171429,
1171449,
1171469,
1171489,
1171509,
1171529,
1171549,
1171569,
1171589,
1171609
],
"core.event_count": 26,
"core.first_event_number": 1171109,
"core.last_event_number": 1171609,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110_reco_stage1_20250908T154435_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346276.0,
"core.end_time": 1757346276.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2232_df-s03-d3_dw_0_20250907T150110.hdf5"
}
]
} 