OctoInformBeta

OpenAI preprint, September 2026 · §5. Disjoint windows of entropy gain

§5 (setting): suffix laws and entropy windows: When tags fix signed slopes, with compactly supported terminal probabilities and positive integral scale ratios

Finite conditional suffixes give decreasing entropy windows.

Hover over a blue or green term in the text, or a node in the graph, to find it in the other. Select it to read its declaration both in Lean and in words.

The Setting

Conditioning on the tags of a finite word gives affine mixtures of a terminal measure. Removing successive letters leaves a family of suffixes, each viewed at the scale inherited from its removed prefix. Their entropy windows compare the entropy at two spatial scales.

A weighted affine iterated function system on a finite alphabet 𝒜\mathcal A consists of maps φi\varphi_i and weights pip_i satisfying (1), with arbitrary real translations tit_i and signed slopes rir_i. φi(x)=rix+ti,0<|ri|<1,pi>0,∑i∈𝒜pi=1.\varphi_i(x)=r_i x+t_i,\qquad 0<|r_i|<1,\qquad p_i>0,\qquad \sum_{i\in\mathcal A}p_i=1.(1)✦ The complete affine composition of a finite word ww is the pair (rw,tw)(r_w,t_w), and its word similarity is fwf_w, defined by (2); the first letter is outermost. (r⌀,t⌀)=(1,0),(riw,tiw)=(rirw,ritw+ti),fw(x)=rwx+tw.\begin{gathered} (r_\varnothing,t_\varnothing)=(1,0),\qquad (r_{iw},t_{iw})=(r_i r_w,r_i t_w+t_i),\\ f_w(x)=r_w x+t_w. \end{gathered}(2)

For a tagging map κ:𝒜→ℬ\kappa:\mathcal A\to\mathcal B and a symbol a∈𝒜a\in\mathcal A, the tag fiber Fκ(a)F_\kappa(a) is the set of symbols with the tag of aa.✦ The fiber mass mκ(a)m_\kappa(a) for a tagging map κ:𝒜→ℬ\kappa:\mathcal A\to\mathcal B and a symbol a∈𝒜a\in\mathcal A is the total weight of the tag fiber of aa; the conditional symbol probability pκ(v∣a)p_\kappa(v\mid a) for v∈𝒜v\in\mathcal A is the normalized weight specified in (3). Fκ(a)={v∈𝒜:κ(v)=κ(a)},mκ(a)=∑v∈Fκ(a)pv,pκ(v∣a)={pv/mκ(a),κ(v)=κ(a),0,κ(v)≠κ(a).\begin{gathered} F_\kappa(a)=\{v\in\mathcal A:\kappa(v)=\kappa(a)\},\qquad m_\kappa(a)=\sum_{v\in F_\kappa(a)}p_v,\\ p_\kappa(v\mid a)= \begin{cases} p_v/m_\kappa(a),&\kappa(v)=\kappa(a),\\ 0,&\kappa(v)\ne\kappa(a). \end{cases} \end{gathered}(3) For reference and sampled words w,v∈𝒜nw,v\in\mathcal A^n, the conditional word probability Qw(v)Q_w(v) is the product of the conditional symbol probabilities.✦ The conditional tail law TwμT_w\mu associated with a reference word w∈𝒜nw\in\mathcal A^n and a measure μ\mu on ℝ\mathbb R is the affine mixture of μ\mu with weights Qw(v)Q_w(v) over v∈𝒜nv\in\mathcal A^n, as in (4), where f#μf_\#\mu denotes pushforward. Qw(v)=∏k=0n−1pκ(vk∣wk),Twμ=∑v∈𝒜nQw(v)(fv)#μ.Q_w(v)=\prod_{k=0}^{n-1}p_\kappa(v_k\mid w_k),\qquad T_w\mu=\sum_{v\in\mathcal A^n}Q_w(v)\,(f_v)_\#\mu.(4) Only the nn specified tags are conditioned on; the future represented by μ\mu remains unconditioned.

Entropy Between Scales

For a measure ν\nu on a measurable space XX, a labeling map f:X→Bf:X\to B and a label b∈Bb\in B, the fiber mass mν(f,b)m_\nu(f,b) is the mass of the set labeled bb, and the partition entropy Hν(f)H_\nu(f) is the negative sum of the fiber masses times their logarithms, as in (5), with 0ln⁡0=00\ln0=0. mν(f,b)=ν{x∈X:f(x)=b},Hν(f)=−∑b∈Bmν(f,b)ln⁡mν(f,b).m_\nu(f,b)=\nu\{x\in X:f(x)=b\},\qquad H_\nu(f)=-\sum_{b\in B}m_\nu(f,b)\ln m_\nu(f,b).(5) At scale s>0s>0 and offset u∈ℝu\in\mathbb R, the shifted grid cell index ℓs,u\ell_{s,u} is the integer-valued function on ℝ\mathbb R given in (6). For a measure ν\nu on ℝ\mathbb R, the shifted grid entropy Hs,u(ν)H_{s,u}(\nu) is the partition entropy of ℓs,u\ell_{s,u}, and the averaged grid entropy Hs(ν)H_s(\nu) is its integral over offsets u∈[0,1]u\in[0,1], as in (6). ℓs,u(x)=⌊xs+u⌋,Hs,u(ν)=Hν(ℓs,u),Hs(ν)=∫01Hs,u(ν)du.\ell_{s,u}(x)=\left\lfloor\frac{x}{s}+u\right\rfloor,\qquad H_{s,u}(\nu)=H_\nu(\ell_{s,u}),\qquad H_s(\nu)=\int_0^1 H_{s,u}(\nu)\,du.(6) For a measure ν\nu on ℝ\mathbb R and positive scales s,ts,t, the entropy window Gs,t(ν)G_{s,t}(\nu) is the difference in (7). These entropies use natural logarithms; division by ln⁡2\ln2 gives bits. Gs,t(ν)=Hs(ν)−Ht(ν).G_{s,t}(\nu)=H_s(\nu)-H_t(\nu).(7)✦ A measure μ\mu on ℝ\mathbb R is compactly supported if there is a real B≥0B\ge0 such that μ(ℝ\[−B,B])=0\mu(\mathbb R\setminus[-B,B])=0.✦ For a measure ν\nu on ℝ\mathbb R and a∈ℝa\in\mathbb R, the signed dilation DaνD_a\nu is the pushforward of ν\nu under x↦axx\mapsto ax.

The Claim

Let WW be a finite word of length mm. For j∈ℕj\in\mathbb N, write W<jW_{<j} for its first min⁡(j,m)\min(j,m) letters and W≥jW_{\ge j} for the remaining suffix.

Definition 1. For a finite reference word ww, a terminal measure μ\mu on ℝ\mathbb R and a∈ℝa\in\mathbb R, the scaled conditional tail law Tw(a)μT_w^{(a)}\mu is the measure in (8). Tw(a)μ=Da(Twμ).T_w^{(a)}\mu=D_a(T_w\mu).(8)✦ The scaled suffix law τj\tau_j associated with a finite reference word WW and a terminal measure μ\mu on ℝ\mathbb R is given for j∈ℕj\in\mathbb N by (9). τj=TW≥j(rW<j)μ.\tau_j=T_{W_{\ge j}}^{(r_{W_{<j}})}\mu.(9)✦ For a scaled suffix law τj\tau_j, a scale s>0s>0 and a positive integer NN, the suffix entropy window gjg_j is the quantity in (10). gj=Gs,Ns(τj).g_j=G_{s,Ns}(\tau_j).(10)✦

Theorem 1. Let μ\mu be a compactly supported probability measure, let s>0s>0, and let NN be a positive integer. For every finite reference word WW, its suffix windows satisfy (11). 0≤gj≤ln⁡N(j∈ℕ).0\le g_j\le\ln N\qquad(j\in\mathbb N).(11)✦ If the signed slope is constant on every tag fiber, the windows are nonincreasing as specified in (12). gk≤gj(0≤j≤k).g_k\le g_j\qquad(0\le j\le k).(12)✦

In the paper, the result reads: retaining the environment and constants of §4, with θ\theta the type shift, ω\omega the infinite type environment, μθjω\mu_{\theta^j\omega} its conditional tail law, νθjω,n\nu_{\theta^j\omega,n} its conditional block law, and RjR_j the signed prefix dilation, put τ̃j=DRjμθjω\widetilde\tau_j=D_{R_j}\mu_{\theta^j\omega}, so that conditional independence gives τ̃j=(DRjνθjω,n)*τ̃j+n\widetilde\tau_j=(D_{R_j}\nu_{\theta^j\omega,n})*\widetilde\tau_{j+n}; choose δ>0\delta>0 with Δv(μ)/ln⁡2≥δ\Delta_v(\mu)/\ln2\ge\delta for every v>0v>0, where Δv(μ)=ln⁡2−Gv,2v(μ)\Delta_v(\mu)=\ln2-G_{v,2v}(\mu), and an integer M>4M>4 with 5/M<δ/25/M<\delta/2, and for an integer L≥1L\ge1 put r=2−Lr=2^{-L} and g̃j=Gr/M,Mr(τ̃j)/ln⁡2\widetilde g_j=G_{r/M,Mr}(\widetilde\tau_j)/\ln2, obtaining 0≤g̃j≤2log⁡2M0\le\widetilde g_j\le2\log_2 M and g̃j≥g̃j+1\widetilde g_j\ge\widetilde g_{j+1}.

What Is Checked

The bounds concern the scaled suffix laws of a finite weighted affine system with any tagging map, a compactly supported terminal probability measure, a finite reference word, a positive scale and a positive integer scale ratio; monotonicity additionally assumes that symbols with the same tag have the same signed slope.