Jobsub ID 36336.148@dunegpschedd02.fnal.gov
| Jobsub ID | 36336.148@dunegpschedd02.fnal.gov |
| Workflow ID | 2323 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:53:52 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2025-09-08 15:52:30 |
| From worker node | Hostname | cn002.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:09:28 |
| Input files | vd-protodune:np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5592s) |
| CPU time | 1h (4813s = 86%) |
| Max RSS bytes | 3064594432 (2922 MiB) |
| Outputting started | 2025-09-08 15:42:41 |
| Output files | |
| Finished | 2025-09-08 15:52:30 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
at call=53 anode=0
[16:40:34.914] D [ glue ] <FrameFanin:nfsp> EOS at call=53 with 8
[16:40:34.914] D [ glue ] frame sink sees EOS
[16:40:34.914] D [ pgraph ] <Pgrapher:> graph execution complete
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.31 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.25 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.91 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.55 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.3 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.16 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.98 sec
[16:40:34.914] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.8 sec
[16:40:34.914] I [ timer ] Timer: WireCell::Aux::Resampler : 0.43 sec
[16:40:34.914] I [ timer ] Timer: WireCell::Aux::Resampler : 0.43 sec
[16:40:34.914] I [ timer ] Timer: WireCell::Aux::Resampler : 0.43 sec
[16:40:34.914] I [ timer ] Timer: WireCell::Aux::Resampler : 0.42 sec
[16:40:34.915] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[16:40:34.915] I [ timer ] Timer: WireCell::Gen::Retagger : 0.02 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:34.915] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:40:34.915] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:34.915] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:40:34.915] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:40:34.915] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:40:34.915] I [ timer ] Timer: Total node execution : 62.06000071205199 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 30374 traces tagged "gauss"
FrameSaver: q=4.80625e+06 n=601107 tag=gauss
wclsFrameSaver: saving 36893 traces tagged "wiener"
FrameSaver: q=5.05496e+06 n=572887 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 989 XUs and 752 XVs -> 32 XUVs
C:0 T:1 169 XUs and 146 XVs -> 7 XUVs
C:0 T:2 384 XUs and 351 XVs -> 18 XUVs
C:0 T:3 198 XUs and 203 XVs -> 8 XUVs
C:0 T:4 545 XUs and 648 XVs -> 21 XUVs
C:0 T:5 290 XUs and 352 XVs -> 19 XUVs
C:0 T:6 155 XUs and 243 XVs -> 8 XUVs
C:0 T:7 301 XUs and 338 XVs -> 6 XUVs
C:0 T:8 3870 XUs and 5157 XVs -> 1253 XUVs
C:0 T:9 860 XUs and 700 XVs -> 41 XUVs
C:0 T:10 1083 XUs and 1500 XVs -> 58 XUVs
C:0 T:11 802 XUs and 764 XVs -> 57 XUVs
C:0 T:12 19 XUs and 24 XVs -> 1 XUVs
C:0 T:13 3793 XUs and 3943 XVs -> 152 XUVs
C:0 T:14 663 XUs and 753 XVs -> 52 XUVs
C:0 T:15 4038 XUs and 5175 XVs -> 294 XUVs
2027 XUVs total
784 collection wire objects
2027 potential space points
Neighbour search...
223313 tests to find 98376 neighbours
Iterating with no regularization...
Begin: 2.88704e+08
0 2.7544e+08
1 2.74863e+08
2 2.74816e+08
Now with regularization...
Begin: 2.70552e+08
0 2.70533e+08
08-Sep-2025 16:41:25 BST Closed output file "np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924_reco_stage1_20250908T154125_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 129.376 201.912 295.851 201.331 43.6072 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.7784e-05 0.000122997 0.000227674 0.000105217 4.13318e-05 27
produce:tpcrawdecoder:PDVDTPCReader 12.4042 26.9731 68.512 24.812 10.0658 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.03285 0.0361534 0.0576737 0.0334245 0.00591247 27
produce:pdvddaphne:DAPHNEReaderPDVD 5.36601 7.15214 8.98332 7.39235 0.834606 27
produce:ophit:OpHitFinder 0.035794 0.0674718 0.0795174 0.0695034 0.00979193 27
produce:opflash:OpFlashFinderVerticalDrift 0.0112956 0.0197115 0.0289082 0.0209979 0.00498802 27
produce:wclsdatavd:WireCellToolkit 57.3363 72.5331 107.819 73.2209 10.9294 27
produce:gaushit:GausHitFinder 0.878493 1.5762 2.16207 1.6313 0.380686 27
produce:nhitsfilter:NumberOfHitsFilter 0.000395246 0.000572776 0.000772842 0.000559624 9.36428e-05 27
produce:reco3d:SpacePointSolver 5.82024 12.2416 20.4631 12.5832 3.38151 27
produce:hitpdune:DisambigFromSpacePoints 0.108291 0.296067 0.712416 0.288175 0.127897 27
produce:pandora:StandardPandora 25.221 73.9179 135.219 72.252 29.6089 27
produce:pandoraTrack:LArPandoraTrackCreation 0.881583 2.09569 6.08808 2.05365 0.986899 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0206331 0.0411434 0.0684988 0.0424266 0.0108 27
[art]:TriggerResults:TriggerResultInserter 2.3942e-05 5.48261e-05 0.000168464 5.2264e-05 2.49747e-05 27
end_path:out1:RootOutput 5.705e-06 1.1593e-05 5.4096e-05 9.06e-06 8.85525e-06 27
end_path:out1:RootOutput(write) 3.04546 4.88251 6.93487 5.16109 0.878641 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4950.23 MB
Peak resident set size usage (VmHWM): 3064.59 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924_reco_stage1_20250908T154125_keepup.root
\tHists: np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924_reco_stage1_20250908T154127_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924_reco_stage1_20250908T154125_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924_reco_stage1_20250908T154125_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346087.0,
"core.end_time": 1757346087.0,
"core.events": [
883018,
883038,
883058,
883078,
883098,
883118,
883138,
883158,
883178,
883198,
883218,
883238,
883258,
883278,
883298,
883318,
883338,
883358,
883378,
883398,
883418,
883438,
883458,
883478,
883498,
883518,
883538
],
"core.event_count": 27,
"core.first_event_number": 883018,
"core.last_event_number": 883538,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924_reco_stage1_20250908T154127_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346087.0,
"core.end_time": 1757346087.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1687_df-s05-d4_dw_0_20250907T034924.hdf5"
}
]
} 