+j^~ VdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl m Z ejeZehdZejdZejd Zejd Zd ed e e eeffd Zde e eefd e e eeffdZd efdZd@de ed e eeffdZ d@dede ed e e eeffdZ d@dede ed e efdZdadadZ dAdede ede e eefd e efdZ!dBdeded efdZ"de#d e$fd Z%ejd!Z&ejd"ej'Z(d ed e e eeffd#Z)d ed efd$Z*de#d e e#fd%Z+d&ed efd'Z,d(ed e e eeffd)Z-d*e e#d+e e eefd dfd,Z.d*e e#d e e/fd-Z0d.ed e e1fd/Z2d0e e1d1e e1d e e e/fd2Z3d*e e#d3e e/d+e e eefd dfd4Z4 d@d5d6d7dede ed8e/d9e$d e e#f d:Z5d5d6d7d;ed8e/d9e$d e e#fd<Z6d5d6d7d=e e#d8e/d9e$d e e#fd>Z7d@de ed e e#fd?Z8dS)Cu LightClaw history — read session transcripts and return history messages. Mirrors: src/history/ (session-store.ts, message-parser.ts, session-reader.ts, text-processing.ts). Storage note: since Hermes "spec 002" the message transcript lives in SQLite (``~/.hermes/state.db`` → table ``messages``), not in per-session ``.jsonl`` files. ``sessions.json`` survives in both eras as a ``session_key → session_id`` index. To stay compatible with BOTH the new and the legacy builds we read in two tiers (see ``read_session_history``): 1. SQLite via :class:`hermes_state.SessionDB` (new builds). 2. Per-session ``.jsonl`` transcript files (legacy builds, or when SessionDB is unavailable / returns nothing). N)Path)AnyDictListOptional)LOCALFILE_SCHEME>customsession model_changethinking_level_changeu<用户发送了文件:\s*(?P.+?)\s*\((?P[^)]+)\)zT\[media attached:\s*(?P\S+)\s*\((?P[^)]+)\)\s*\|\s*(?P[^\]]+?)\s*\]z---[0-9a-f-]+\.textreturnc|sgSg}t|D]d}||d|dded}t |D]}|d}|d}|t |kr|||d<|||d<nXd|vr|dd d n|}t d |pd }||||d |d z }|S)uRecover file attachment info from a previously-rendered message body. Returns a list of ``{name, mimeType?, size?, uri?}`` dicts, ordered as in TS ``extractFileAttachments``: first pass extracts entries from ``"用户发送了文件: ..."`` descriptions, then a second pass enriches them (in order) with the ``[media attached: ...]`` markers' MIME/URI fields, finally appending unmatched markers as standalone entries. namesize)rrrmimeurimimeType/r.file)rrr) _USER_FILE_DESC_REfinditerappendgroupstrip_MEDIA_ATTACH_RElenrsplit_TRAILING_HASH_REsub)rfilesm media_idxrrbases ;/home/agentuser/.hermes/plugins/lightclawbot/src/history.pyextract_file_attachmentsr)7s  "$E  ( ( . .88 aggfoo3355ggfoo335577 8 8 8 8I  & &t , ,  wwv$$&&wwu~~##%% s5zz ! !+/E) Z (&)E) U # #-0CZZ3::c1%%b))SD$((d33=vD LL$DEE F F FQ Lr$c2i}|D]r}|dpd}||vrC||}dD]7}||s ||r ||||<8`t|||<st|S)zCMerge duplicate file entries by name (mirrors TS deduplicateFiles).r)rrr)getdictlistvalues)r$seenfrexistingks r(deduplicate_filesr5Ys&(D !!uuV}}" 4<<DzH0 ' '||A'15588'"#A$HQK 'aDJJ   r*ctjdptjd}tj|dS)z-Return the default Hermes sessions directory. HERMES_HOMEz ~/.hermessessions)osenvironr-path expanduserjoin) hermes_homes r(_default_sessions_dirr?ls@*..//R273E3Ek3R3RK 7<< Z 0 00r* sessions_dirc|p t}tj|d} t |dd5}t j|}dddn #1swxYwYt|tr|SnE#tt j tf$r&}t d||Yd}~nd}~wwxYwiS)u,Load sessions.json → {session_key: entry}. sessions.jsonrutf-8encodingNz(Could not load sessions.json from %s: %s)r?r9r;r=openjsonload isinstancer.FileNotFoundErrorJSONDecodeErrorOSErrorloggerdebug)r@dr;r2dataexcs r(load_session_storerSrs/-//A 7<<? + +DL $g . . . !9Q<.jsonl`` transcript file (legacy builds). Fallback chain (mirrors session-store.ts): 1. entry.sessionFile (absolute, or relative to sessions_dir) 2. /.jsonl 3. sessions.json sibling directory (offline / downloaded sessions) Returns ``None`` when no transcript file exists (e.g. on new SQLite-only builds, which is the expected case there). N.jsonl session_file sessionFilerB)r?r-r9r;isabsr=isfiledirname) r]r@rYrP jsonl_namesfpp2sessions_json_path sibling_dirp3s r(resolve_transcript_pathrvsP t/-//A&&&J  YY~ & & B%))M*B*B  gmmB''@RW\\!R-@-@Aw~~a    a $ $B w~~b a99'//"455K{a'' W\\+z 2 2 7>>"   I 4r*r,contentrolecvt|tr|St|tsdSg}|D]r}t|ts|dd}|dvr@|dd}t|tr||sd|S)z0Extract plain text from a message content field.r,type)r output_text input_textr )rJstrr/r.r-rr=)rwrxpartsrYttxts r( _extract_textrs'3 gt $ $rE""%&&   IIfb ! ! 5 5 5))FB''C#s## " S!!! 99U  r*msgc|dpddkrdSt|dd}tt jd|S)z?Return True if a user-role message is actually system-injected.rxr,userFrwz,^System:\s*\[\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2})r-rVrboolrematchr)rrs r(_is_system_injectedrsk 2$$&&&00u B// 0 0D H$**,,WW X XXr*z4\[(?P[^\]]+)\]\((?Plocalfile://[^)\s]+)\)z1^[`\"']?MEDIA:\s*(?P(?:~/|/)\S+)[`\"']?\s*$cd|r t|vrgSdt|DS)zAExtract ``[name](localfile://...)`` links from assistant replies.cdg|]-}|d|dd.S)rr)rr)r).0r%s r( z,_extract_localfile_links..+sD    88   r*)r _LOCALFILE_MD_REr)rs r(_extract_localfile_linksr'sI #4//   !**400   r*crd|vr|Sdtjdtfd}t||S)uReplace ``MEDIA:/path`` lines with ``localfile://`` Markdown links. Transcript JSONL stores the raw AI reply including MEDIA: tags (the framework strips them at delivery time via ``extract_media()``). When serving history back to the front-end, we convert them to the same ``📎 [name](localfile:///path)`` format used by live delivery so the user sees clickable download links. zMEDIA:r%rc|dd}tj|p|}d|dt |dS)Nr;z "'`,.;:)}]u📎 [z]())rrrstripr9r;basenamer )r%r;rs r(_replacez2_convert_media_tags_to_localfile.._replace=sdwwv$$&&--m<<w%%-999 09$9999r*)rMatchr~ _MEDIA_TAG_REr#)rrs r( _convert_media_tags_to_localfiler1sPt :BH:::::   Xt , ,,r*c|dd}|sdSddddddd}||d}t|d |}|d }t|tt fr|nd}|s|d vrdS||d }|||d <|dkr$t |}|rt||d <n+|dkr%t|} t| } | r| |d <|S)zuConvert a raw JSONL message object to a HistoryMessage dict. Returns None if the message should be skipped. rxr,Nr assistantsystemtool)rrrr toolresult tool_resultrw timestamp)r)rxrwr$) r-rVrrJintfloatr)r5rr) rrole_rawrole_maprxr raw_timestamprresultr$converted_textlocalfile_linkss r(normalize_messagerEsY wwvr""H t  H <<((+ 6 6D ++T 2 2DGGK((M!+MCBB  .-F7O Mr*transcript_pathcl|dr|dtd dzS|dzS)uReturn the sidecar usage path for a given transcript jsonl path. Convention: ``/.jsonl`` → ``/.usage.jsonl``. riN .usage.jsonl)endswithr )rs r(_resolve_usage_log_pathr}sD ))B/#h--/0>AA ^ ++r*r;ctj|sgSg} t|dd5}|D]}|}|s t j|}n#t j$rY@wxYwt|tsZ| d}|t|ts| d}t|ttfs| | dt||d dddn #1swxYwYn5#t$r(}td ||gcYd}~Sd}~wwxYw|S) uParse the usage sidecar jsonl into a list of entries. Each entry has shape:: {"roundMsgId": str, "timestamp": int, "usage": dict | None} A ``usage`` of ``None`` denotes a *placeholder* line written by ``outbound._persist_turn_usage`` for turns where the tracker had no delta to report. Placeholders are kept in the list to preserve ordinal alignment with transcript turn-end assistants — the attach step skips them rather than writing ``null`` onto a message. Malformed lines are skipped silently. A missing file returns ``[]`` — equivalent to "no usage data yet" (e.g. fresh sessions before the feature shipped). rCrDrEusageNr roundMsgId)rrrz#Failed to read usage sidecar %s: %s)r9r;rmrGrrHloadsrLrJr.r-rrrrMrNrf) r;entriesr2linestrippedobjrtsrRs r(_read_usage_logrs" 7>>$   $&G $g . . . !  ::<<*X..CC+H!#t,,(($Zt-D-D$WW[))!"sEl33"%'',"7"7"%b''"'  %                0 >ED>E E8E3-E83E8messages usage_entriesc|sdSt}|sdSfd|D}d|D}t|t|krtd|Drrtd|DrYt||}t |D]7\}}|||d}|&|||d<8dSt ||dS)uIn-place: attach each usage entry to its matching turn-end assistant. "Turn-end assistant" = the last ``assistant`` of each turn (a ``user`` message opens a turn; tool/system rows are transparent), mirroring openclaw's "one usage per turn, on the turn's last assistant". Matching strategy — **absolute-time anchoring** (preferred): Each sidecar entry carries the ``stop_typing`` timestamp of its round; each turn-end assistant carries its store timestamp. We anchor every entry to the turn-end assistant *nearest in time* via an order- preserving (monotonic) minimum-cost assignment. Because inter-turn gaps (tens of seconds) dwarf the intra-turn writer skew (sub-second), each entry lands on its own turn — and, crucially, the pairing stays correct when the two sides differ in count (a command round wrote no entry, or a real-but-zero-token round was skipped). That count drift is exactly what pure ordinal alignment cannot survive: one missing or extra entry shifts every subsequent pairing. Fallback — **ordinal tail-alignment**: Used when timestamps are unusable on either side, or there are more entries than turn-ends (legacy data). Pairs the i-th entry with the i-th turn-end counting from the end, so freshly-added turns at the tail line up and pre-feature turns at the head go without usage. Placeholder entries (``usage is None``, written when a real turn could not be measured) take part in matching so they *claim* their own slot and keep neighbours from drifting onto it, but no payload is attached — a null usage would mislead the renderer into showing "0 tokens" instead of "unknown". Nc`g|]*}t|d+Sr _usage_ts_msr-)rirs r(rz-_attach_usage_to_messages..s1 V V VQl8A;??;7788 V V Vr*cRg|]$}t|d%Srr)res r(rz-_attach_usage_to_messages..s,HHHQ QUU;//00HHHr*c3K|]}|duV dSNrrs r( z,_attach_usage_to_messages..s&..! ......r*c3K|]}|duV dSrrrs r(rz,_attach_usage_to_messages..s&00! 000000r*r)_collect_turn_end_assistantsr all_align_entries_by_time enumerater-_attach_usage_tail_ordinal) rrturn_end_assistantsend_tsentry_ts assignment entry_posend_posrs ` r(_attach_usage_to_messagesrsFB 6x@@  V V V VBU V V VFHH-HHHH Mc"56666 ..v... . . 7 00x000 0 0 7,Hf== "+J"7"7 D D Iw!),0099E}>CH(1 27 ; ;x)>%1#**+ABBB)-&  ;  %& ")""#9::: r*rct|tst|ttfsdS|dkrt|dznt|S)aNormalise a timestamp to epoch milliseconds, or ``None`` if unusable. Transcript rows store epoch *seconds* (e.g. ``1780387827.02``) while the usage sidecar stores epoch *milliseconds* (``int(time.time() * 1000)``). A value below ``1e12`` is therefore treated as seconds and scaled up, so both sides become directly comparable without the callsite assuming a unit. ``bool`` is rejected explicitly (it is an ``int`` subclass in Python). NgmBg@@)rJrrr)rs r(rrsT"d:b3,#?#?t!#d599v  b 9r*rrc   t|t|c} |dkrgStd fdt|dzD}t dzD] }d|d|<td|dzD]u}t| dzD]_}|||dz }||dz |dz t||dz ||dz z z}||kr|n||||<`vdg|z}| }}|dkr}|dkrw|||dz }||dz |dz t||dz ||dz z z}||kr|dz ||dz <|dz}|dz}n|dz}|dkr|dkw|S)uOrder-preserving minimum-cost assignment of entries onto turn-ends. Returns a list parallel to *entry_ts*: ``result[i]`` is the index into *end_ts* that entry ``i`` is anchored to (strictly increasing across i), or ``None`` if unassigned. Cost is absolute time distance; the chosen turn-ends form an increasing subsequence — matching the fact that sidecar entries and turn-end assistants share one chronological order while the head of the transcript may pre-date the sidecar feature. Caller guarantees ``len(entry_ts) <= len(end_ts)`` so a full assignment of every entry exists. Complexity ``O(p * q)`` with both small. rinfc"g|] }gdzz S)rr)r_rqs r(rz*_align_entries_by_time..:s# 0 0 0a3%1q5/ 0 0 0r*rgN)r rrangeabs) rrrqdpjrskiptakerrrs @@r(rr$s x==#f++DAqAvv ,,C 1 0 0 0 05Q<< 0 0 0B 1q5\\1a 1a!e__55q!a% 5 5Aa5Q.jsonl`` transcript file — used when SessionDB is unavailable (older builds) or yields no rows. In both cases the ``session_id`` comes from the surviving ``sessions.json`` index. z9No sessions.json entry for session_key=%s sessions_dir=%sr]r^r,z*Entry for session_key=%s has no session_idNT)include_ancestorsz)@)@    <((HEIIk,B,BHbJ C[QQQ    B ~ ::d;LL   NNNC    LLLLLL   -EYH  ;$9$;$;W\\! -H-H-HII / ; ;  G-h FFF   ?  O H    #:|U C CD  +      LL>EEE $g . . . !&&((C                94EEE !EYGGGH  '(?(E(EFF  ? %h > > >     7s         OsB44 C&>C!!C&>AE FE>>F0H/H# H/#H''H/*H'+H// I!9II!I!7.J&& K0KKrcg}|D]}|}|s tj|}n#tj$rYAwxYw|dt vrb|ddkr|t|dtr|dn|}|ddkrt|rt|}||r |ddvr| |t||kr || dn|S) z:Parse a legacy JSONL transcript into HistoryMessage dicts.rzrx session_metamessagemodelzdelivery-mirrorNrr) splitlinesrrHrrLr-_SKIP_LINE_TYPESrJr.rrrr )rrrrrparsedr normalizeds r(rrs~H  #$#$zz||   Z%%FF#    H  ::f  !1 1 1  ::f   / / (2&**Y2G2G'N'NZfjj###TZ 777  0 0 0  s # #  &s++     F+3HHH  #### #H  5 58UFGG  8CsAAArc0g}|D]r}t|ts|dr.t|r>t |}|P|r |ddvr]||st ||kr || dn|S)aNormalise SQLite conversation rows into HistoryMessage dicts. ``raw_messages`` come from ``SessionDB.get_messages_as_conversation`` in the OpenAI ``{role, content, tool_calls?, ...}`` shape (content already decoded and sanitised). We reuse :func:`normalize_message` for file-attachment / ``MEDIA:`` link recovery, and drop delivery-mirror echoes (``observed``) plus system-injected user turns to match the previous JSONL behaviour. observedNrxr)rJr.r-rrrr )rrrrrrs r(rr sH$$#t$$   77:     s # #  &s++     F+3HHH  #### #H  5 58UFGG  8Cr*ct|}g}|D]\}}|s|dp|dpd}|s7|dp|d}|dp|d}|||||d|d d |S) z/List all sessions from the sessions.json index.r]r^r, updated_at updatedAt display_name displayName) sessionKeyr^rrc0|dpdS)Nrr,)r-)xs r(zlist_sessions..HsaeeK006Br*T)keyreverse)rSrWr-rsort)r@rXrr rYsidupdateddisplays r( list_sessionsr4s | , ,EFkkmm  U  ii %%E;)?)?E2  ))L))CUYY{-C-C))N++Guyy/G/G  "        KK66KEEE Mr*r)NN)r,)9__doc__rHloggingr9rpathlibrtypingrrrrconfigr getLogger__name__rN frozensetrcompilerrr"r~r)r5r?rSr[r_rbrcrgrvrr.rrr MULTILINErrrrrrrrrrrrrrrrrrr*r(rs"  ,,,,,,,,,,,,$$$$$$  8 $ $9 RZC 2:" BJ12234S#X+?D T$sCx.1 d4S>6J    &1s1111  Xc] d38n    6#'   3- d38n    $#'3-c]2 6#'&*(((3-( DcN #(c] ((((^3c3$YTYdYYYY2:;  8L 34S#X+?-3-3----(141HTN1111p,S,S,,,,3#3$tCH~"63333l>M4j>MS#X'>M >M>M>M>MB4:$s)0 :S :Xe_ : : : :*5k* K* (3-****Z54j5c5S#X'5 55556#'l llll3-l  l  l  $Z lllld -D-D-D -D -D -D  $Z -D-D-D-Df $D$D$Dt*$D $D $D  $Z $D$D$D$DV dr*