Network backpressure
Feed exactly what a socket provides. Do not wait for read boundaries to align with records.
Partial streams, explained honestly
Framing can be incremental. Trust cannot.
ParcelStream can accept arbitrary chunks and emit complete parcels. A complete sd1 envelope still has to verify before any of those bytes become trusted.
Playground 02 · chunk simulator
Move the read boundary.
Change the chunk size and watch headers and values arrive in inconvenient pieces. Only complete parcels appear in the event log.
Encoded bytes—
Reads—
Parcels—
Read tape
Decoder events
Where it earns its keep
Useful in four different ways.
Incremental framing is modest, but broadly reusable.
Large verified payloads
After envelope verification, walk a payload in bounded pieces and hand complete values to application code.
Unknown type skipping
A length lets a parser skip an application type it does not understand without interpreting the bytes.
Concatenation
Complete streams can be joined without a new wrapper, count, or re-encoding pass.
Two different questions
| Question | Answer |
|---|---|
| “Do I have a complete parcel?” | The incremental decoder can answer from type + length + received bytes. |
| “May I trust or act on this parcel?” | Only an authenticated boundary can answer. For an sd1 payload, verify the complete envelope first. |
No partial trustA parser returning a complete value means its framing is complete. It says nothing about sender identity, purpose, expiry, or tampering.
Incremental JavaScript
import { ParcelStreamDecoder } from "./signed-data.mjs";
const decoder = new ParcelStreamDecoder({
maxValueBytes: 8 * 1024 * 1024
});
for await (const chunk of response.body) {
for (const [type, value] of decoder.feed(chunk)) {
handleCompleteParcel(type, value);
}
}
decoder.finish(); // rejects a partial trailing header/valueUse this loop directly only when the transport already authenticates chunks. For bytes carried inside sd1, first verify the envelope, then feed slices of envelope.payload.
Incremental Ruby
serializer = SignedData::Serializers::ParcelStream.new
decoder = serializer.decoder(max_value_bytes: 8 * 1024 * 1024)
socket.each_chunk do |chunk|
decoder.feed(chunk.b).each do |type, value|
handle_complete_parcel(type, value)
end
end
decoder.finishThe Ruby and JavaScript tests run the same examples at every possible split point, including one-byte reads, empty values, binary values, and explicit size ceilings.
When raw streaming is safe
ParcelStream can travel over TLS, QUIC, an authenticated file, a signed chunk protocol, or another transport with its own integrity boundary. In those settings, the stream decoder is simply a compact record framer.
Without such a boundary, code may inspect or stage complete parcels but must not treat them as authenticated claims. This is the same distinction as peek: readable is not trusted.
A future signed-stream profile would be separate
True early trust needs each chunk or checkpoint to carry authentication: for example chained signatures, a Merkle root with proofs, or independently signed records. That changes replay, ordering, truncation, and finalization rules.
- Give it a distinct protocol/profile identifier.
- Specify ordering, omission, duplication, and final-chunk behavior.
- Authenticate type, position, and total/continuation metadata.
- Publish adversarial vectors before calling it interoperable.
It should not be smuggled into sd1 by teaching a whole-envelope signature to mean partial verification.
The design decision
SignedData core: complete portable envelope and verification rules.
ParcelStream tool: optional serializer and incremental framer shipped alongside the runtimes.
Application profile: owns parcel type meanings and decides whether another authenticated transport permits early use.
Future signed streams: a separate, explicitly versioned design if a real use case needs early authenticated delivery.