justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262872.0@dunegpschedd01.fnal.gov

Jobsub ID262872.0@dunegpschedd01.fnal.gov
Workflow ID10977
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-03 20:53:42
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2025-12-03 21:52:30
From worker nodeHostnamedune001-542727.0-lcg2529.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-03 20:54:41
Input filesusertests:000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root
JobscriptExit code0
Real time57m (3450s)
CPU time56m (3368s = 97%)
Max RSS bytes7426572288 (7082 MiB)
Outputting started2025-12-03 21:52:13
Output files
Finished2025-12-03 21:52:30
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

AFM detsim jobscript.
Input PFN = root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
[20:55:54.034] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:54.048] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:54.061] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:54.075] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:54.089] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:54.102] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
03-Dec-2025 20:55:54 UTC  Initiating request to open input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"
03-Dec-2025 20:55:56 UTC  Opened input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"
Begin processing the 1st record. run: 20000031 subRun: 0 event: 29 at 03-Dec-2025 20:55:58 UTC
SimDepoSource got 5377768 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 5377768 depos spanning: [0.0432409, 2.02045e+06]us
[21:46:57.457] W [  gen   ] <IncoherentAddNoise:addnoiseapa7> model EmpiricalNoiseModel:empericalnoisemeganodes produced 6174 samples instead of expected 6000, future warnings muted 
Retagger: tagging trace set: wiener with 132440 traces, 132440 summary
Retagger: tagging trace set: gauss with 108647 traces, 0 summary
wclsFrameSaver: saving 108647 traces tagged "gauss"
FrameSaver: q=7.21823e+07 n=2274458 tag=gauss
wclsFrameSaver: saving 132440 traces tagged "wiener"
FrameSaver: q=7.90787e+07 n=2091445 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
03-Dec-2025 21:51:46 UTC  Opened output file with pattern "000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_205449Z.root"
03-Dec-2025 21:52:06 UTC  Closed output file "000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_205449Z.root"
03-Dec-2025 21:52:06 UTC  Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      3354.34       3354.34       3354.34       3354.34          0            1     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0409324     0.0409324     0.0409324     0.0409324         0            1     
simulate:tpcrawdecoder:WireCellToolkit          3345.37       3345.37       3345.37       3345.37          0            1     
simulate:rns:RandomNumberSaver                0.000552201   0.000552201   0.000552201   0.000552201        0            1     
[art]:TriggerResults:TriggerResultInserter    8.0882e-05    8.0882e-05    8.0882e-05    8.0882e-05         0            1     
end_path:out1:RootOutput                       3.224e-05     3.224e-05     3.224e-05     3.224e-05         0            1     
end_path:out1:RootOutput(write)                 8.92318       8.92318       8.92318       8.92318          0            1     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9536.83 MB
  Peak resident set size usage (VmHWM): 7426.57 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
outfile is 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_205449Z.root
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w10977s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 29,
        "art.last_event": 29,
        "art.process_name": "detsim",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v10_11_00d01",
        "core.data_stream": "out1",
        "core.data_tier": "detector-simulated",
        "core.end_time": 1764798724.0,
        "core.event_count": 1,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 29,
        "core.last_event_number": 29,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1764798706.0
    },
    "parents": [
        {
            "did": "usertests:000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"
        }
    ]
}
===================
.:
total 793936
-rw-r--r-- 1 dune001 dune 812838640 Dec  3 21:52 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_205449Z.root
-rw-r--r-- 1 dune001 dune    116731 Dec  3 21:52 wcls-sim-drift-simchannel-nf-sp.log
-rw-r--r-- 1 dune001 dune      8169 Dec  3 21:52 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_2025-12-03T_205449Z.log
-rw-r--r-- 1 dune001 dune      7101 Dec  3 21:52 jobscript.log
-rw-r--r-- 1 dune001 dune      1141 Dec  3 21:52 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_205449Z.root.json
-rw-r--r-- 1 dune001 dune       985 Dec  3 21:52 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_205449Z.root.ext.json
-rw-r--r-- 1 dune001 dune       224 Dec  3 20:54 all-input-dids.txt
-rw-r--r-- 1 dune001 dune       193 Dec  3 21:52 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune         0 Dec  3 20:55 debugprod.log
justIN time: 2025-12-19 15:42:00 UTC       justIN version: 01.05.03